Big data can extend and enrich a data warehouse, but cannot replace it. It is not data warehouses that are dead, but the traditional way of designing and building them.
Data warehouse automation is one of those inventions that is so early on and so innovative that only early adopters and visionaries have taken the leap of faith. The rest of the business world has either not heard of automation or mistrusts something that looks too good to be true. But just because it looks too good to be true doesn’t make it false.
A monk writing out copies of the bible with his quill might have thought a typewriter was too good to be true, who knows what trickery he would have imagined was contained in the subsequent development of word processors, computers and iPads?
The printing press modernised and revolutionised the world of print. Now data warehouse automation is revolutionising data warehousing with similar results; financial savings, gains in speed, efficiency and accuracy. But before we look at the future of data warehousing, let’s go back to the eighties when data warehouses were created, to see why it’s time to move on.
The problems of traditional data warehouses
The traditional way of building data warehouses is snail slow. ETL coding is written by hand so it takes months to build the data warehouse which is often out of sync with requirements by the time of deployment. It’s like a man taking years to restore his two-seater convertible to have it ready just as his wife announces she’s pregnant with twins.
The sad fact is that the true value and capabilities of the data are rarely understood until the data warehouse is built, but by then it is too late. The warehouse, if it isn’t abandoned before completion, is an expensive, inflexible disappointment.
Perhaps this is why many technologists and thought leaders are ready to declare the data warehouse dead – no longer relevant in the age of big data. But these prognosticators are mistaken.
Getting your data warehouse right
At its core, the data warehouse integrates critical and valuable enterprise data that is not found in big data sources and that continues to be the primary data resource for descriptive, prescriptive and decision analytics. It serves as corporate memory, collecting the body of history that makes time-series and trend analysis possible.
The data warehouse also organises and structures data to make it understandable and useful for consumption by many different business stakeholders. This business intelligence gives organisations the edge, making them more competitive, more customer focused, more profitable.
Data warehouse automation, the future of the data warehouse
Data warehouses will be needed for the foreseeable future, but they need to quicker to build and at reasonable cost, readily adapt to changing requirements and be responsive to business and technical change. And all of this has to occur without compromising solution quality.
Enter data warehouse automation, the future of the data warehouse. Data warehouse automation delivers quality and effectiveness through the ability to build better solutions; solutions which meet real business requirements.
With data warehouse automation the business can make changes much later in the development process and change can occur more frequently with less disruption, waste and rework. This efficiency is not only a joy, it saves time, resources and money. In a traditional data warehouse build, it is especially difficult to get complete and correct requirements due to the linear development process. Automation also brings quality benefits through standards enforcement and standardising the development processes.
The agility of the automated data warehouse is not limited to its ability to change in the warehouse development process, it can also handle changes in business requirements. Responding to change in real time and without the delay of lengthy projects is the essence of business agility.
By: Rob Mellor, General Manager - Mainland Europe, WhereScape