BI Built to Order, On-Demand: Automating Data Warehouse Delivery

| January 23, 2015

This week, Dr. Barry Devlin published a provocative new paper on data warehouse automation – “BI Built to Order, On-Demand: Automating Data Warehouse Delivery.”

You can grab it here, if you’re curious. And you should be. Because in the paper Devlin does two things: first, he considers a few Inconvenient Truths about how data warehouses are built and managed – or misbuilt and mismanaged – and, second, he makes the case for data warehouse automation as a common-sense fix for today’s often mismanaged data warehouse development.

When Devlin described his vision for the business information system he called a “data warehouse” – back in early 1988 – we just didn’t have the tools to efficiently design, build, and manage warehouse systems. Everything, or almost everything, had to be done by hand: there weren’t any ETL tools, data integration suites, studios, platforms or workbenches. But even once we got primitive versions of these tools – starting in 1993 or thereabouts – things didn’t magically get better. In fact, by 2003, we were already starting to come to grips with the empirical fact that data warehouse projects took too long to build, failed to deliver on many of the promises Devlin had outlined in his paper, and, most important, were too hard to change. We know: WhereScape-the-company grew out of the integration experiences of our founders, who specialized in fixing just these problems.

But a great point that Devlin makes is that most of these problems were byproducts of what might be called an “out of phase” development process. Simply put: building data warehouse systems was and to some degree still is a disintegrated affair. In larger organizations, it is performed by separate teams or groups of developers, each working with their own set of tools, each using their own methodology, and each building at their own pace. According to Devlin, this is one of the biggest impediments to traditional analytic development.

“Modeling, database design and development of population routines required multiple, disconnected iterations involving business users, modelers, database administrators and ETL programmers at different times, each using different and unconnected tools. These gaps and tool transitions slowed the process and gave rise to design errors and inconsistencies,” Devlin writes.

The upshot is that this model compromises both the consistency of data and the timeliness of application delivery. Devlin sees data warehouse automation software – which centralizes data warehouse and analytical development in a single tool – promoting an iterative, agile development methodology, and implements a shared metadata repository – as a prescriptive Rx for this problem.

“The common environment and shared metadata repository offered by data warehouse automation overcomes this … by integrating the design and delivery of the data model, database structure, and the population process in one place – whether for a warehouse or mart,” he writes. “All the design and population metadata is stored together in a single repository, allowing development to flow smoothly and iteratively from user requirements, through database design, to creation of population routines. By integrating all the steps of the design and development process, consistent and quality data can be delivered quickly to the business for immediate review and early acceptance.”

Data warehouse automation software isn’t a turnkey fix. Devlin recognizes this. All the same, it’s a way to eliminate out-of-phase development, centralize the development process, and enforce a consistent, delivery-focused development paradigm. It gives you a solid foundation on which to build your data warehouse. Data warehouse automation software has other benefits that aren’t at all confined strictly to development. As Devlin notes, it promotes collaboration between business and IT, making it possible to produce data-driven – or business-data-driven – apps.

I’ll say more about this in a follow up post.

Common Data Quality Challenges and How to Overcome Them

The Importance of Maintaining Data Quality Improving data quality is a top priority for many forward-thinking organizations, and for good reason. Any company making decisions based on data should also invest time and resources into ensuring high data quality. Data...

What is a Cloud Data Warehouse?

As organizations increasingly turn to data-driven decision-making, the demand for cloud data warehouses continues to rise. The cloud data warehouse market is projected to grow significantly, reaching $10.42 billion by 2026 with a compound annual growth rate (CAGR) of...

Developers’ Best Friend: WhereScape Saves Countless Hours

Development teams often struggle with an imbalance between building new features and maintaining existing code. According to studies, up to 75% of a developer's time is spent debugging and fixing code, much of it due to manual processes. This results in 620 million...

The Competitive Advantages of WhereScape

After nearly a quarter-century in the data automation field, WhereScape has established itself as a leader by offering unparalleled capabilities that surpass its competitors. Today we’ll dive into the advantages of WhereScape and highlight why it is the premier data...

WhereScape 3D 9.0.4 Now Available: Integrate with Microsoft Purview

We are excited to announce the release of WhereScape 3D Version 9.0.4, which is packed with new enhancements, highlighted by the integration with Microsoft Purview. Additional features include advanced data profiling for custom connections, Pebble extensions for...

Related Content

Common Data Quality Challenges and How to Overcome Them

Common Data Quality Challenges and How to Overcome Them

The Importance of Maintaining Data Quality Improving data quality is a top priority for many forward-thinking organizations, and for good reason. Any company making decisions based on data should also invest time and resources into ensuring high data quality. Data...

Common Data Quality Challenges and How to Overcome Them

Common Data Quality Challenges and How to Overcome Them

The Importance of Maintaining Data Quality Improving data quality is a top priority for many forward-thinking organizations, and for good reason. Any company making decisions based on data should also invest time and resources into ensuring high data quality. Data...

Developers’ Best Friend: WhereScape Saves Countless Hours

Developers’ Best Friend: WhereScape Saves Countless Hours

Development teams often struggle with an imbalance between building new features and maintaining existing code. According to studies, up to 75% of a developer's time is spent debugging and fixing code, much of it due to manual processes. This results in 620 million...