Documentation is an essential component of any sustainable data architecture, yet for practical reasons, it is often neglected by teams still working with manual processes. Developers are under constant pressure to meet deadlines and are judged more on the code they produce than how well they document it.
The Need for Automated Documentation
Pseudo Agile strategies are now common in data warehousing, and we often hear of teams working in sprints with SCRUM meetings and so on. However, most developers still type out every line of code by hand and work with 90s ETL technology. While the deadlines and iterations they have to meet are set on Agile timeframes, the tools and methodologies they use are not Agile, which puts them at a huge disadvantage even before they start. It is rare that sufficient consideration is given to documentation when sprints are defined. Under these constraints, developers will always be under stress and need to take shortcuts to deliver code. Documentation is always the first thing to be neglected or dropped completely.
The need to write documentation inhibits productivity in a number of ways. Firstly, if you take regular breaks to document what you have done, you break your flow and write less code. If you leave it until you have finished, you forget what you have done. Even leaving it until a Monday to document last week’s code means you will forget a decent amount of what you did. The more productive you were, the more you have to forget.
Data Automation for Sustainable Architectures
In the short-term world of delivering code just in time for the next deadline, documentation is seen as a nice-to-have and may not even be examined by those responsible for data governance. It is only further down the line, when our architectures are suffering from clutter and are hard to maintain, or one of the developers leaves the company and someone comes in to replace them, that we realize the value of accurate, standardized documentation.
Data Automation enables us to be flexible and switch target databases much easier than in the past. A choice of database is no longer a ten-year decision. This also means that we no longer have to rip up and start again if we want to switch modeling styles. The code we write now can be tweaked depending on needs, and can be sustainable for many years. For this reason, it needs to be reliable to stand the test of time, which also applies to how it is documented. Architectures need to supersede individuals and databases, so when one developer leaves, the code must pass on to colleagues seamlessly.
Automated Documentation Benefits
WhereScape removes the documentation issue from a developer’s schedule entirely, enabling them to focus on higher-value tasks and meet Agile deadlines without cutting corners. All they have to do is click one button and all their work will be documented to a level of detail that would take many hours to do by hand.
WhereScape is metadata-driven. The WhereScape GUI is a simplified manifestation of all the metadata that sits behind it and makes the data warehouse work as it is shown to. This out-of-the-box documentation feature means that each action taken by users and every item and structure within the architecture is stored, such as:
- The code itself.
- Lists of columns and objects, who created them, whether they are included in certain jobs and so on.
- Transformations.
- Lineage backwards and forwards (where did data come from and where did it go after this point?).
- Data types and all information about the current object you are looking at.
- Interactions between the various objects within the architecture.
WhereScape documentation is a roadmap that goes back through the project you have been working on, providing hyperlinks to every stage of the process so you can click in and see its code and structure. All the documentation that developers should be writing by hand, but often don’t for the various reasons outlined above, is generated automatically at a much higher level of detail. To do this for an entire architecture would be months of work; too much to do whilst maintaining productivity in writing code.
Improve Developer Productivity
Such comprehensive documentation means bugs can be caught and fixed much faster. Rather than having to go through badly documented code by hand to find the fault, WhereScape highlights faulty code in red. As well as finding the problem quickly, this prevents you from working on good code that is mistakenly diagnosed as incorrect.
“The turnaround of our bugs has been incredible. It takes two hours now where it used to take us two weeks. The autonomy the developers have over deployments means a quicker turnaround for performing re-tests.” – James Gardiner, Data Warehouse Technical Lead at Admiral Group.
When discussing Data Automation, it is important to remember that human creativity is still paramount. It is augmented by automation tools that eradicate repetitive manual work, enabling developers’ brains to be less fatigued and so more creative. However, in many areas of data warehousing, creativity and flair are undesirable and even harmful. Documentation is exactly the kind of mundane, repetitive work suited to automation. All we need is a reliable account of what was done in a language that other developers can understand, nothing more, nothing less.
"The documentation is generated which saves us a lot of time and it's always accurate." – Marleen Gerbraad, Delivery Manager Datalogistics at Rabobank.