There’s a trend in IT firms. CEOs and evangelists like to try and coin the next major buzz phrase with the word ‘data’ in it. There’s big data, fast data, data analytics, data provenance, time-series data and then there are data lakes, data marts and the data warehouse.
TechTarget defines the data warehouse as federated repository (either physical or logical) for all the data that an enterprise's various business systems collect.
Data warehouse, in simple English
To use a simple retail analogy, specific groups of users or jobs lead to the creation of data markets (we will call them ‘data marts’)… and in the same way that markets are fed by a central distribution and storage warehouse, individual data marts all come together to reside in a bigger data warehouse.
But the notion of (or at least the operation of) the data warehouse is broken claims Michael Whitehead, president & founder of data warehouse automation company WhereScape.
Smell something fishy? A company that sells software tools designed to put intelligence controls into data warehousing environments says that traditional data warehousing approaches are flaky. Is this just a platform to spin WhereScape wares, or does Whitehead have a point?
Industry failure, linguistic gymnastics
"It is time to call it, the data warehouse has failed. It was a great idea, some of the concepts were right, yet as an industry we failed to deliver on its promise," asserts Whitehead.
We might find some proof if we look at ‘magical’ analyst house Gartner. The wizards in residence are busy renaming their quadrilateral cauldron as follows:
- In 2014 it was Magic Quadrant for Data Warehouse Database Management Systems.
- In 2015 it was Magic Quadrant for Data Warehouse and Data Management Solutions for Analytics.
- In 2017 we will enjoy the Magic Quadrant for Data Management Solutions for Analytics.
WhereScape’s Whitehead says that there are any number of other examples of linguistic gymnastics by vendors and analysts i.e. people are distancing themselves from the term as a whole.
“While the industry is willing to kill the data warehouse as they have not delivered, there is no viable alternative. The industry (like the people, if you believe in democracy) are right, and data warehouses should be killed. But it should have been toppled by a better alternative, not collapsed all by itself with no natural successors,” says Whitehead.
Pointing to growing data management mavericks in the new world (from Qlik and Tableau through to Alteryx, Paxata, Trifacta et al), Whitehead suggests that these are worthy players, but none of them engage with the boring side of data governance, scalability, reuse, persistence, lineage and history that are still required if we are to rely on data (and keep CEOs out of prison).
These so-called ‘boring’ elements are among the elements that WhereScape seeks to address. This then is where that word ‘automation’ comes in again i.e. putting the tools in place to make these boring but essential things happen automatically based on defined logic.
So that’s data warehouses sorted then, for now, sort of. They (the warehouses) need more automation for overall data management in the face of what are both technical and legal concerns.
Next up to put the frighteners on data management and the likelihood of cloud computing even continuing is this thing we call data relevance.
“Data led organisations have no idea how good their data is. CEOs have no idea where the data they get actually comes from, who is responsible for it etc. yet they make multi million pound decisions based on it. Big data is making the situation worse not better. So in the world of democratization of data, people get what they ask for, not what they necessarily need or want (think Brexit?) and are unconcerned with the consequences,” claims WhereScape’s Whitehead.
Oh goodness, non-relevant un-automated badly warehoused cloudy big data in a Brexit style maelstrom of democratized openness. Wasn’t cloud computing supposed to make things easier? Yes, but not quite yet, hang in there please.
The article was originally published in Forbes.