The six-month project has now been completed, smashing the previous three-year forecast by using agile development to overhaul each data warehouse business area in ‘sprints’.
Tesco PLC is the second-largest retailer in the world and a global company operating in dozens of countries. Tesco is facing new business challenges and opportunities, and is especially concerned about how the company manages its discount programs. Tesco needed a global view of its data to coordinate management information.
An Aging Data Warehouse
At the heart of management reporting, the data warehouse acts as the official system-of-record for Tesco. The company had been working on a long-term development plan to renovate the data warehouse using traditional waterfall methodologies. In the meantime, dozens of ungoverned databases sprung up, acting as separate sandboxes to generate specific outputs, all without a blueprint.
Andy Ruckley, Head of Technology: Data & Information Platforms, initiated a review of Tesco’s management information systems, including infrastructure, architecture, data models, hardware, processes, interactions, and business uses for data. Ruckley found many pockets of excellence. He also found problems that are typical of older data warehouses. There were over a hundred playpens (or sandboxes) where data was dump
A New Approach Needed
Over the years, the data warehouse would periodically run out of capacity. The usual solution was to buy newer equipment, which was an enticing option when funding was available. Once installed, the logical and physical data architecture of the old data warehouse was simply lifted-and-shifted onto the new platform.
Ruckley concluded that Tesco should avoid long project cycles with big development teams. Instead, Tesco needed to start fresh with the right data model across all functions, spanning all regions worldwide. A new approach was also needed to renovate the data warehouse incrementally, enabling business value had to be delivered continually throughout the development process.
Ruckley found several companies who had successfully pursued agile data warehouse development, allowing them “to turn around projects more quickly and model the business more accurately.” Ruckley was particularly interested in “how companies transitioned their data warehouse into a fit-for-purpose capability,” using the Teradata platform with WhereScape’s data warehouse automation tools.
Key Pain Point
One of the company’s greatest pain points is the amount of money Tesco spends on promotions. Was Tesco gaining or losing money on promotions? The company wanted additional insights into “how products were selling by brand, location, and price point” from a global perspective. When Ruckley showed management the prototype of the promotion analysis, they became engageand wanted to use the prototype immediately.
Ruckley initiated a proof-of-concept consisting of a single sprint, which went well. A plan was formulated to conduct the initial project from February to May of 2014. The strategy was to put a “working solution” in front of business users every four weeks. Three teams of six to seven persons performed sprints on four-week cycles. The results of one team’s sprint were used as the input to the team in next track.
Each team in the three tracks specialised on performing specific functions of Tesco’s Discovery- Build-Release layers:
- Data Driven Design layer: The Discovery phase is “scouting ahead” to lead the team to a business solution. WhereScape 3D was the primary tool to accomplish this function.
- Integrated Data layer: The Build phase is where you remove the shortcuts and do proper modeling and indexing to generate the SQL Data Definitions. WhereScape RED was the primary tool, in combination with the Teradata system to test the new data definitions.
- Access/Presentation layer: The Build phase is where the data is made available to the business users. MicroStrategy is the primary tool, with a minor role by WhereScape.
- WhereScape enabled the sprint teams to manage the data architecture and model key concepts, while sharing the current status with version control. “WhereScape 3D is used to profile the data, while WhereScape RED does everything else, such as lineage and ETL. We generated more than 300 views from the business policies.”
- Agile methodology requires quick turnarounds for trying and testing alternatives. The combination of WhereScape and Teradata provided this responsive infrastructure. “WhereScape allows us to model a concept, press GO, generate the DDL code, and test the physical structure as to whether it actually works. And, it is all documented! WhereScape adapts well to agile. You can easily change things. It does not impose limitations on the agile process.”
- Data architecture design is a blend of art and science, with many shades of gray. WhereScape was used as a learning tool that enhanced the skill level of their team members. “Some had never used the WhereScape tool. WhereScape allowed us to quickly create a model and then test it, so that we could learn the implications of design decisions. We are continuing to learn. Other tools do not allow for the quick cycles necessary to enable this learning.”Ruckley concluded that Tesco should avoid long project cycles with big development teams. Instead, Tesco needed to start fresh with the right data model across all functions, spanning all regions worldwide. A new approach was also needed to renovate the data warehouse incrementally, enabling business value had to be delivered continually throughout the development process.