There are many aspects to big data – the ‘V’s as they are known.  Volume is one (how much data do you actually have), but variety (different types of data, such as office files as well as relational data), the data’s velocity (how fast are you receiving it and how fast does the next stage of the process require it) are others.  Alongside this are veracity (how good is the data?) and value (what are we getting out of the holding, aggregating, analysing and reporting of the data anyway?).

As a fairly simple yet effective example of how one organisation used big data to create a new service, consider Westminster (London) city’s use of sensors in parking bays tied into an app on smartphones that allow drivers to find parking bays that are empty.  This needed to pull together lots of different data sets (those from the sensors in the parking bays, GPS for the cars searching for a parking bay, payment for parking, etc).

Another example is how DEFRA (UK Department of Environment, Food & Rural Affairs) has had to put more investment into flood planning and response.  It is one thing taking pure meteoroidal data and saying “it is going to rain heavily here – there’s a chance of a flood”, and being prepared for it. By working with an IBM-incubated company (KnowNow Information), a system has been built that not only uses predictive analytics using the meteorological data, but also uses sensors on the ground around flood-prone areas so as to better sense when upstream and up-valley flows of water are becoming dangerous.  It enables the right types of emergency equipment to be placed in the right place before it is needed, rather than trying to respond when it is already too late – or by spending too much money in putting in place flood defences where they are not actually needed.

As an example of a company that just wouldn’t exist if it wasn’t for its capability to deal with big data, look at Tesla.  Its vehicles are a complex and tightly enmeshed set of monitors, sensors and actuators, all of which are sending information within the car and outside of it back to Tesla HQ - all needing real-time management.  It is through this that Elon Musk is able to say that the crashes with Teslas are or are not down to the car itself, or down to the way the driver was reacting at the time. Having the right approach and technological platform in place, Tesla has been able to incrementally improve its car for drivers, through things like its “Ludicrous” mode and its semi-autonomous driving capability, neither of which were available on the vehicle to start off with.

A lot of these examples are generally down to someone having that flash of inspiration and being able to find the right company to partner with.  Increasingly, though this need for the super-intelligent spark of genius is having to be commoditised through the provision of the right kind of tools to identify, aggregate, normalise, analyse and report on the data available.  This requires the right tools to be chosen – such as WhereScape - to automate the identification and aggregation of data that can then be better moved through the analytical process and reported to the right people at the right time.

It is also where we see Hadoop and noSQL come into play – as well as the old guard of Oracle, IBM, Teradata and co for data handling, along with systems such as HDS Pentaho, QlikView and Tableau for data analytics and reporting.

Quocirca believes that it is now a race to identify, gain and acquire the right tools that enable an organisation to unleash the power of its existing employees who are business, not technology, focused.  This needs data automation tools; it needs ones that operate at the right velocity and add the right levels of value to the business across its variety and volumes of data that it has to deal with. This also requires many existing data sets to be both integrated and migrated – another area where tools such as WhereScape can help.

Clive Longbottom, Analyst, Quocirca.

Clive Longbottom is the co-founder and service director at Quocirca and has been an ITC industry analyst for more than 25 years.