In spring of 2010, I began working on a new venture with WhereScape’s client, a procurement organization for a major restaurant chain. The project was developing an Enterprise Data Warehouse (EDW) in a Data Vault “flavor”. The organization had a new management team which understood the benefits of using an agile methodology, so it perfect for me as I had always felt that this organization was well positioned to benefit from adopting an agile approach to data warehousing.
The organization was already a Wherescape RED customer, and already knew all of its great features. While adopting agile techniques, using scrums or even using Wherescape RED is not “the” answer, it can certainly be part of the answer – and they were open to its use.
The key principles of Agile data warehousing are tightly aligned with WhereScape Pragmatic Data Warehousing™ (PDW) methodology. The following shows the PDW methodology principles in conjunction with the Scrum method that the organization adopted at the beginning of the new EDW.
1. Divide and conquer
A big bang tactic to data warehousing usually ends in disaster. One of the main principles of Scrum is to divide the project (Product) into smaller 2-4 weeks chunks (Sprints) and even smaller 24-hours pieces. “Simplicity--…--is essential”. Projects in Wherescape RED are easily divisible by business area, “star”, “uow”, even individual Data Warehouse objects.
2. Adjust to business users' needs
WhereScape RED “welcomes changing requirements, even late in development”. Instead of adopting a strict change management process, WhereScape adapts the Agile approach to change management where the company’s stakeholders can easily change their minds according to their changing business needs. Scrum’s relatively short Sprints allow the team to adjust to business’ needs more easily.
3. Support the team
“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done”. Building a team that was not just a collection of individuals working under the same management was one of the immediate benefits of Scrum. We, as teammates, select our own tasks (or choose to work together on tasks). The tasks we choose are small, taking up to a day to complete.
4. Customer satisfaction
Everyone on the team adopted the main goal of agile methodology: “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”. At the daily 15-minute stand-up meeting, each of us talked about what we worked on the previous day, what is planned for today, impediments , and what kind of help we need from a business owner to complete the task.
5. Deliver working components regularly
In other words, “a potentially shippable product at the sprint end”. This way the business community sees and uses the results of the DW/BI development very early and supports, including financially, further iterations of the EDW. Wherescape RED’s elaborate mechanism of versioning and deployment works well here.
6. Fail fast
If a Sprint fails, the failures are fast and cheap. Scrum/ Wherescape RED allows dismissing the development iteration midstream. The team recovers and generates another Sprint in virtually no time.
7. Short timeframes
Strive for iterations of one to two weeks. Iterations of this length provide more opportunity to govern the project effectively due to the greater feedback provided by regular delivery and business users’ input “Deliver working software frequently … with a preference to the shorter timescale.”
8. Involve users early
This is the only way to keep them focused and involved. The business community is a key stakeholder on any DW/BI project. The sooner you involve them, the sooner you find out what their requirements are. The product owner is one of the pillars of the Scrum: “Business people and developers must work together daily throughout the project.”
9. Test throughout the lifecycle
Quality Assurance (QA) is an integral part of every Sprint to avoid common data quality challenges. Testing is embedded into the WhereScape methodology. “Delivered outcome must be correct”.
10. Embrace standards
Embrace common development standards via WhereScape RED functionality and company-adopted governing software. This includes both end-user and technical documentation sets auto-generated by WhereScape RED. Now our client goes all the way with burn-down charts, elaborate storyboards, pivotal tracker, and other highly visible attributes of Scrum.
The bottom line is that now, months into the project, Agile/Scrum/WhereScape RED have blended very well, truly helping the client in reaching its main goal of building a high quality, reliable and well-documented EDW. For more information on the WhereScape RED product, please visit wherescape.com
Note: All the italicized quotations are from the agile manifesto.