This is one of those really good ideas I wish I had come up with.
In the last couple of months, I've met with two customers who independently hit on the same idea: using WhereScape's RED and 3D in a hackathon!
It might seem like a stretch, but when you think about it, it makes a heck of a lot of sense.
After all, WhereScape RED is a data warehouse and big data automation tool. You use it to build, test, and optimize a data system. It automates all of the tedious work that should already have been automated in the first place. WhereScape 3D is a scoping and prototyping tool. You use it to peer into upstream systems, to identify relevant data sources and data structures, and to build prototypes of a data warehouse system. Okay, you say, but what does this have to do with hacking?
Well, nothing. A hackathon doesn't have to have anything to do with hacking. It's an exercise in which software developers collaborate with user interface (UI) and user experience (UX) designers, project managers, and (in WhereScape's case) line-of-business people – to see how much they can accomplish in a finite period of time. That said, hackers do frequently make use of automation technology – like NMAP and similar automated port scanning or probing tools – that have legitimate info-sec uses. Once a hacker finds an open or potentially vulnerable system, she uses her creativity and intelligence to try to break that system. The automated tooling does the tedious, boring, time-consuming heavy lifting for her. Sound familiar? It should. It's making smart use of automation.
And making smart use of automation is the WhereScape Way.
Anyway, back to our customers and their hackathons. For starters, their implementations were quite different. One was more genteel – or, at any rate, less hectic and high energy. The other was a classic hackathon: kickoff at 3 PM, a steady flow of Red Bull and pizza – punctuated by timely servings of craft IPA – with participants hacking on through to 3 PM of the following day.
By the sounds of it, both hackathons were highly successful. This isn't all that surprising, given the productivity boost you get from automation – and, naturally, the quality of the people involved.
It's the simplicity of the thing that's its genius. In a hackathon, the one resource you don't have an abundance of is time. You've got 24 hours, no more, no less. Anything that can enable you to iterate more quickly – to define and develop software units of work, to automate routines to test and, if possible, to break them, to automatically document what you've done, and, finally, to start on a new sprint – gives your participants a decided advantage. This is a great example of the intelligent use of automation. It's all about letting the software do the heavy lifting while human beings deal with the problems that require creativity, ingenuity, deep analysis, and, yes, problem-solving synthesis.
Yes, a hackathon is an artificial environment, but it's one that's designed to closely resemble – to model – the real world. In the real world, people are impatient for results. They always already want it yesterday. In the real world, IT groups almost never have enough time and money. They're always already being asked to do more with less. The hackathon is a race against time, pure and simple.
It's about producing the best possible deliverable given what you have to work with.
A few other similarities come to mind, too.
First, in both cases, you need to focus on what you can realistically achieve. In a hackathon, you have to be highly specific, because time is so compressed. In conventional data warehouse, data hub, data vault/lake development, you don't need to be quite so specific. This doesn't mean you should abandon specificity, however.
Instead, you need to choose a problem that's big enough to make an impact, but which is contained enough such that it can still be completed in the time that's available – whether that's one night, one week, or one month. Lean software development has a concept called Minimal Viable Product, or MVP. Think of MVP as a results-oriented construct for balancing risk versus reward in software development. We've all been involved in software projects that were dominated from start to “finish” by the specter of risk. The goal wasn't so much to deliver a great project but to not screw up. Agile and lean software development practices evolved in response to precisely this problem.
A concept like MVP forces you to clearly define what you want to build – its scope, its constitutive elements, its most essential attributes – in such a way as to maximize what you can realistically achieve while (at the same time) minimizing risk. In lean software development, the MVP is a function of time, money, and other finite resources. If the deliverable is too big or too complex to be completed in the time that's available, your product – or your team's part of the product – will not be viable. From the perspective of your line-of-business customers, you will have failed.
Second, you need to spend or allocate your resources wisely. Irrespective of whether those resources are time, money, or intangible capital, they're limited; how you allocate or dispose of them will make the difference between success and failure. The issue isn't having good ideas; it is rather prioritizing the right ideas – i.e., identifying the ideas that are essential to the main opportunity or MVP. In a sense, it's about shelving good ideas in favor of better or more germane ones.
Third and last, you need to take advantage of all the help you can get. That's why automation is essential. By making use of common-sense automation technologies, you (and your team) won't have to do nearly as much work. What's more, the work you do will be of much greater value. It'll be work that taxes the creativity, ingenuity, and perspicacity of the team you've assembled.
A hackathon might not be a good idea for every organization – although don't underestimate the soft benefits of inter- and intra-team bonding that can occur! – but the mindset of achieving a meaningful outcome in a finite or constrained period of time is something that's hard to argue against. The hackathon brings this problem into startlingly clear focus. It's also an excellent demonstration of the value of common-sense automation in software development. Finally, if I can toot WhereScape's own horn, it's likewise a demonstration of how useful great software tools are!