Having been in the technical arena for a very long time (I’ve actually used punch cards), I’ve assumed that everyone knows about and wants to use automation. Whether coding, system administration, or data warehouse development, automation has always been the way to get the job done easier, faster, and with better results. My job now focuses on Data Warehouse Automation (DWA) – designing, developing, deploying, and operating data warehouses using the WhereScape solutions.
But I’ve realized that many organizations and IT departments still haven’t fully adopted DWA. Some think it is too rigid. It “destroys creativity.” It doesn’t apply in your very unique environment. “I can write code better than some automation tool!” Let me start by explaining some non-obvious benefits of DWA and then address some people's objections.
Data Warehouse Automation
Automation, whether it’s technical (like WhereScape), in manufacturing, post office sorting, or any other field, does two things: it reduces errors, and it increases consistency. Someone looking to adopt DWA, rather than hand-coding or piece-mealing their data warehouse, immediately thinks they will get to the finish line faster. And they will. But what’s not obvious is how DWA changes the race, and continues after the race is over.
DWA can have EXTREME improvements in productivity. Without the hand-offs and waiting on others, without introducing typing errors, and being able to “fail fast,” you can do in minutes what otherwise might take hours or days. I can sit down with a developer who might be using Yellowfin, Tableau, or PowerBI and make changes to the development environment right in front of them. Do they want a new calculated column added? Maybe they want a transform changed so NULL values in a column are no longer NULL. With WhereScape I can make those changes in under a minute and update the data warehouse right in front of them. They can then immediately use their tool of choice to query that new structure and give immediate feedback. That can drastically change how work is done as the development effort becomes a much more collaborative process with the business.
Automate ELT
The second benefit is what happens after the data warehouse is ‘complete.’ Nothing is ever finished. New data sources get added, new transforms are required, and government regulations add new requirements. The less obvious benefit of WhereScape is that all development is done in a single style. Changes are faster and easier because you don’t have to spend time figuring out how some long-gone consultant did their work. You can describe changes through wizards and immediately implement them. In hand-coded environments, modifications take longer and longer as the system grows and more and more errors are introduced. With WhereScape’s DWA, changes are fast and efficient.
Do I Really Want Data Warehouse Automation?
Now, what about the objections that some people have? “Automation of any kind is too rigid.” WhereScape’s use of templates allows you to change how automation works. If you have specific industry, regional, or regulatory requirements, you can implement those in the templates. You will reap the benefits of DWA while meeting any requirements.
An objection may be “DWA destroys creativity.” DWA certainly removes a lot of mundane and repetitive tasks. The creativity required to come up with a solution is still there. It’s just at a more abstract level! Your solutions to your problems are implemented as templates, not as hundreds of lines of hand-written code.
Every environment is unique. Every business thinks that they have issues, restrictions, or requirements that no one else has. And, to some extent, that’s true. WhereScape, because it is template-driven, allows you to modify not just the data warehouse but the entire WhereScape development environment. You can add fields to wizards, restrict options, and change the way WhereScape is used to suit your unique organization.
Automated Coding
Lastly, I’ve heard complaints for decades about how automation, in any field, is generating sloppy code, and someone can hand-write better code. I’m sure that they can. However, writing code is only part of a project. As a programmer, I spent 25% of my time writing code, 50% debugging it, and 25% maintaining something that someone else had written. Can I write better code than generated code? Probably. Can I handwrite tens or hundreds of thousands of lines of code that is clear, documented, and works the first time? No. So, someone can probably write better code, but it won’t be as maintainable or easy to modify as generated code. Writing code is only part of a larger solution.
Building a Data Warehouse with WhereScape
Data Warehouse Automation with WhereScape is an incredible technology. You can finish data warehouse projects in months instead of years, and weeks instead of months. You’ll find that your development productivity increased 4x or 8x or 12x. I know one company that spent two months just talking about how to implement a Type 2 slowly changing dimension. With WhereScape, I can generate that dimension from a source system in under a minute. And it works. And all of the Dimensions will work the same way.
I encourage you to check out WhereScape to see how we can significantly reduce the time, effort, and risk in designing and implementing your data warehouse. We’ll be glad to have a no-pressure talk with you about the benefits of Data Warehouse Automation.
Patrick O’Halloran, WhereScape Product Marketing Manager, has worked in many roles and in many industries, including as a DBA and Data Warehouse consultant. He has broad technical and business experience in developing and delivering enterprise-scale applications.