Menu Request Demo

Data warehousing, big data and the Higgs particle

01 October 2012

Michael Whitehead and Dr. Ralph Schuster

This is a blog I have been wanting to write for a while, but it has only (finally) come about because Ralph has joined the WhereScape lunchtime runs. That’s Dr Ralph Schuster (one of the leading lights in the WhereScape development team) with a doctorate in physics – handy when you want to blog about the Higgs boson. Even more handy when he actually writes most of the blog for you.

This blog is really about the value of a data warehouse, but in keeping with current accepted wisdom, it is about Big Data and the value of a data warehouse. It starts with the Large Hadron Collider (LHC) – the holy grail of big data. The LHC generates more than 20 PB of data per year…and that’s after keeping only 1 per cent of the recorded data.

But this is not the only link between data warehouses (and big data) and the LHC. There is a deeper connection on a more conceptual level. As every educated nerd (or human) knows, one of the main purposes of the LHC is to find the Higgs particle, the last missing building block in standard particle physics, which has been hiding from experimental detection for almost 50 years. The Higgs mechanism is the inevitable ingredient of the standard model, the most successful theory of the fundamentals of matter and forces. The incorporation of the Higgs mechanism is necessary from the very beginning of the formulation of the fundamental equations to give the electroweak bosons mass. The particles which mediate the weak nuclear force responsible for beta decay, the form of radioactivity which, for example, is the source of radiocarbon dating allowing us to estimate the age of archaeological organic objects. Though the indirect effects of the Higgs mechanism are ubiquitous in physics, chemistry, biology and even archaeology the direct evidence is extremely hard to spot and the direct evidence of the existence of a Higgs particle is hardly measurable.

This is where it is like a good data warehouse. The benefits of a data warehouse can be found in almost every business function from finance to fraud detection to human resources, and WhereScape has customers benefiting from data warehouse based analysis from a multitude of verticals including banking, telecommunications, insurance, healthcare, not-for-profits and education institutions. But the accurate measurement of the direct benefits is almost as hard as finding the Higgs particle.

We all know the benefits are there, but we cannot look for them in the data warehouse itself. We need to look for the indirect benefits. And this is where it gets tricky.

“It is the singularly unfair peculiarity of war that the credit of success is claimed by all, while a disaster is attributed to one alone”.

When we look for indirect benefits there are many people who can, quite correctly, claim to be responsible for success, and the business intelligence team is just one. We can provide a great data warehouse, but it needs to be coupled with great analysis, a switched on team who make the right decisions, and an entire organization which acts on it. Each of these groups has legitimate claims for credit.

At WhereScape we recommend looking for opportunities for “assists” that sports concept of the last pass or critical play. There is no call to quantify the exact value of the assist. You don’t allocate assist percentage points to individual team members for a try or a touchdown – you either assisted or you didn’t.

That there is value in fact based decision making is indisputable. Like the Higgs particle, it needs to be there or the basic concept of business decision making fails. Unlike the physicists, we don’t need to spend fifty years looking for it – as long as we don’t look for the glory, and are happy with just being part of the team. So when it comes to the value of the data warehouse, look for the value in overall results. Did we help the organization do well (or less badly)?

And always look for opportunities to claim an assist.


The Works of Tacitus, tr. by Alfred John Church and William Jackson Brodribb, [1864-1877],


Comments 0

Leave a Comment

No comments.