Are you eager to delve deep into the challenges...
Agile Documentation through Automation

Documentation is an essential component of any sustainable data architecture, yet for practical reasons, it is often neglected by teams still working with manual processes. Developers are under constant pressure to meet deadlines and are judged more on the code they produce than how well they document it.
The Need for Automated Documentation
Pseudo Agile strategies are now common in data warehousing, and we often hear of teams working in sprints with SCRUM meetings and so on. However, most developers still type out every line of code by hand and work with 90s ETL technology. While the deadlines and iterations they have to meet are set on Agile timeframes, the tools and methodologies they use are not Agile, which puts them at a huge disadvantage even before they start. It is rare that sufficient consideration is given to documentation when sprints are defined. Under these constraints, developers will always be under stress and need to take shortcuts to deliver code. Documentation is always the first thing to be neglected or dropped completely.
The need to write documentation inhibits productivity in a number of ways. Firstly, if you take regular breaks to document what you have done, you break your flow and write less code. If you leave it until you have finished, you forget what you have done. Even leaving it until a Monday to document last week’s code means you will forget a decent amount of what you did. The more productive you were, the more you have to forget.
Data Automation for Sustainable Architectures
In the short-term world of delivering code just in time for the next deadline, documentation is seen as a nice-to-have and may not even be examined by those responsible for data governance. It is only further down the line, when our architectures are suffering from clutter and are hard to maintain, or one of the developers leaves the company and someone comes in to replace them, that we realize the value of accurate, standardized documentation.
Data Automation enables us to be flexible and switch target databases much easier than in the past. A choice of database is no longer a ten-year decision. This also means that we no longer have to rip up and start again if we want to switch modeling styles. The code we write now can be tweaked depending on needs, and can be sustainable for many years. For this reason, it needs to be reliable to stand the test of time, which also applies to how it is documented. Architectures need to supersede individuals and databases, so when one developer leaves, the code must pass on to colleagues seamlessly.
Automated Documentation Benefits
WhereScape removes the documentation issue from a developer’s schedule entirely, enabling them to focus on higher-value tasks and meet Agile deadlines without cutting corners. All they have to do is click one button and all their work will be documented to a level of detail that would take many hours to do by hand.
WhereScape is metadata-driven. The WhereScape GUI is a simplified manifestation of all the metadata that sits behind it and makes the data warehouse work as it is shown to. This out-of-the-box documentation feature means that each action taken by users and every item and structure within the architecture is stored, such as:
- The code itself.
- Lists of columns and objects, who created them, whether they are included in certain jobs and so on.
- Transformations.
- Lineage backwards and forwards (where did data come from and where did it go after this point?).
- Data types and all information about the current object you are looking at.
- Interactions between the various objects within the architecture.
WhereScape documentation is a roadmap that goes back through the project you have been working on, providing hyperlinks to every stage of the process so you can click in and see its code and structure. All the documentation that developers should be writing by hand, but often don’t for the various reasons outlined above, is generated automatically at a much higher level of detail. To do this for an entire architecture would be months of work; too much to do whilst maintaining productivity in writing code.
Improve Developer Productivity
Such comprehensive documentation means bugs can be caught and fixed much faster. Rather than having to go through badly documented code by hand to find the fault, WhereScape highlights faulty code in red. As well as finding the problem quickly, this prevents you from working on good code that is mistakenly diagnosed as incorrect.
“The turnaround of our bugs has been incredible. It takes two hours now where it used to take us two weeks. The autonomy the developers have over deployments means a quicker turnaround for performing re-tests.” – James Gardiner, Data Warehouse Technical Lead at Admiral Group.
When discussing Data Automation, it is important to remember that human creativity is still paramount. It is augmented by automation tools that eradicate repetitive manual work, enabling developers’ brains to be less fatigued and so more creative. However, in many areas of data warehousing, creativity and flair are undesirable and even harmful. Documentation is exactly the kind of mundane, repetitive work suited to automation. All we need is a reliable account of what was done in a language that other developers can understand, nothing more, nothing less.
“The documentation is generated which saves us a lot of time and it’s always accurate.” – Marleen Gerbraad, Delivery Manager Datalogistics at Rabobank.
See our automated documentation capabilities in action in this video.
Amplifying WhereScape’s Power with Yellowfin: Unveiling New Analytics Opportunities for Your Business
In an age dominated by vast amounts of information, the emphasis on data-driven decision-making has never been greater. The landscape of Business Intelligence (BI) and data analytics has seen a remarkable evolution, emphasizing solutions that can seamlessly integrate...
Data Mesh and Data Fabric: Changing the Game in Data Product Development
Data Mesh vs Data Fabric Data Mesh and Data Fabric are reshaping how organizations approach data product development. In an era where data-driven decisions are central to business success, these innovative paradigms are becoming increasingly crucial. By enabling...
WhereScape Announces the Release of RED 10.0.0.0
WhereScape is pleased to announce the general availability of WhereScape RED 10.0.0.0. This release is the culmination of man-years of effort. It confirms WhereScape’s commitment to continuing to develop new technologies and tools and its commitment to delivering the...
Effective AI through Data Modeling
As we journey deeper into the digital age, the importance of data modeling within the broader landscape of artificial intelligence (AI) has become more pronounced than ever. The success of AI-driven initiatives is tightly woven with the quality and structure of the...
Is Data Vault 2.0 Still Relevant?
TL;DR Yes. Data Vault 2.0 Data Vault 2.0 is a database modeling method published in 2013. It was designed to overcome many of the shortcomings of data warehouses created using relational modeling (3NF) or star schemas (dimensional modeling). Speci fically, it...
Data Vault 2.0 Resources
Data Vault Revisited: A Six-Year Journey into the Secure Data Repository In 2017, Dr. Barry Devlin provided valuable insights about Data Vaults, a concept that sparked interest among businesses and IT professionals. Data Vaults were envisioned as secure repositories...
Understanding Data Vault 2.0
How to Avoid Pitfalls During Data Vault 2.0 Implementation Implementing a data vault as your Data Modeling approach has many advantages, such as flexibility, scalability, and efficiency. But along with that, one must be aware of the challenges that come along with...
Navigating the AI Landscape
The Pivotal Role of Data Modeling In the rapidly evolving digital age, artificial intelligence (AI) has emerged as a game-changer, deeply impacting the business landscape. Its ability to automate operations, refine decision-making processes, and significantly enhance...
Information Management Maturity
Unlocking Your Business Potential: Understanding and Enhancing Information Management Maturity In a recent report by Gartner, they emphasize the crucial role of information in the current business environment, stating, "Through 2025, organizations that are data-driven...
Data Warehousing Best Practices
In modern times, organizations are daily generating huge volumes of data. Appreciating the significance of data, companies are storing data from different departments which can be analyzed to gather insights to help the organization in better decision-making. This...
Related Content

Amplifying WhereScape’s Power with Yellowfin: Unveiling New Analytics Opportunities for Your Business
In an age dominated by vast amounts of information, the emphasis on data-driven decision-making has never been greater. The landscape of Business Intelligence (BI) and data analytics has seen a remarkable evolution, emphasizing solutions that can seamlessly integrate...

Data Mesh and Data Fabric: Changing the Game in Data Product Development
Data Mesh vs Data Fabric Data Mesh and Data Fabric are reshaping how organizations approach data product development. In an era where data-driven decisions are central to business success, these innovative paradigms are becoming increasingly crucial. By enabling...

WhereScape Announces the Release of RED 10.0.0.0
WhereScape is pleased to announce the general availability of WhereScape RED 10.0.0.0. This release is the culmination of man-years of effort. It confirms WhereScape’s commitment to continuing to develop new technologies and tools and its commitment to delivering the...

Effective AI through Data Modeling
As we journey deeper into the digital age, the importance of data modeling within the broader landscape of artificial intelligence (AI) has become more pronounced than ever. The success of AI-driven initiatives is tightly woven with the quality and structure of the...