Agile Documentation through Automation

| February 3, 2021
Agile Documentation through Automation

Documentation is an essential component of any sustainable data architecture, yet for practical reasons, it is often neglected by teams still working with manual processes. Developers are under constant pressure to meet deadlines and are judged more on the code they produce than how well they document it.

The Need for Automated Documentation

Pseudo Agile strategies are now common in data warehousing, and we often hear of teams working in sprints with SCRUM meetings and so on. However, most developers still type out every line of code by hand and work with 90s ETL technology. While the deadlines and iterations they have to meet are set on Agile timeframes, the tools and methodologies they use are not Agile, which puts them at a huge disadvantage even before they start. It is rare that sufficient consideration is given to documentation when sprints are defined. Under these constraints, developers will always be under stress and need to take shortcuts to deliver code. Documentation is always the first thing to be neglected or dropped completely.

The need to write documentation inhibits productivity in a number of ways. Firstly, if you take regular breaks to document what you have done, you break your flow and write less code. If you leave it until you have finished, you forget what you have done. Even leaving it until a Monday to document last week’s code means you will forget a decent amount of what you did. The more productive you were, the more you have to forget.

Data Automation for Sustainable Architectures

In the short-term world of delivering code just in time for the next deadline, documentation is seen as a nice-to-have and may not even be examined by those responsible for data governance. It is only further down the line, when our architectures are suffering from clutter and are hard to maintain, or one of the developers leaves the company and someone comes in to replace them, that we realize the value of accurate, standardized documentation.

Data Automation enables us to be flexible and switch target databases much easier than in the past. A choice of database is no longer a ten-year decision. This also means that we no longer have to rip up and start again if we want to switch modeling styles. The code we write now can be tweaked depending on needs, and can be sustainable for many years. For this reason, it needs to be reliable to stand the test of time, which also applies to how it is documented. Architectures need to supersede individuals and databases, so when one developer leaves, the code must pass on to colleagues seamlessly.

Automated Documentation Benefits

WhereScape removes the documentation issue from a developer’s schedule entirely, enabling them to focus on higher-value tasks and meet Agile deadlines without cutting corners. All they have to do is click one button and all their work will be documented to a level of detail that would take many hours to do by hand.

WhereScape is metadata-driven. The WhereScape GUI is a simplified manifestation of all the metadata that sits behind it and makes the data warehouse work as it is shown to. This out-of-the-box documentation feature means that each action taken by users and every item and structure within the architecture is stored, such as:

  • The code itself.
  • Lists of columns and objects, who created them, whether they are included in certain jobs and so on.
  • Transformations.
  • Lineage backwards and forwards (where did data come from and where did it go after this point?).
  • Data types and all information about the current object you are looking at.
  • Interactions between the various objects within the architecture.

WhereScape documentation is a roadmap that goes back through the project you have been working on, providing hyperlinks to every stage of the process so you can click in and see its code and structure. All the documentation that developers should be writing by hand, but often don’t for the various reasons outlined above, is generated automatically at a much higher level of detail. To do this for an entire architecture would be months of work; too much to do whilst maintaining productivity in writing code.

Improve Developer Productivity

Such comprehensive documentation means bugs can be caught and fixed much faster. Rather than having to go through badly documented code by hand to find the fault, WhereScape highlights faulty code in red. As well as finding the problem quickly, this prevents you from working on good code that is mistakenly diagnosed as incorrect.

“The turnaround of our bugs has been incredible. It takes two hours now where it used to take us two weeks. The autonomy the developers have over deployments means a quicker turnaround for performing re-tests.” – James Gardiner, Data Warehouse Technical Lead at Admiral Group.

When discussing Data Automation, it is important to remember that human creativity is still paramount. It is augmented by automation tools that eradicate repetitive manual work, enabling developers’ brains to be less fatigued and so more creative. However, in many areas of data warehousing, creativity and flair are undesirable and even harmful. Documentation is exactly the kind of mundane, repetitive work suited to automation. All we need is a reliable account of what was done in a language that other developers can understand, nothing more, nothing less.

“The documentation is generated which saves us a lot of time and it’s always accurate.” – Marleen Gerbraad, Delivery Manager Datalogistics at Rabobank.

See our automated documentation capabilities in action in this video.

How to Hire and Retain Data Warehouse Developers

The projected data warehouse developer job growth rate is 21% from 2018-2028, with about 284,100 new jobs for data warehouse developers projected over the next decade, according to Zippia. This surge in demand for data warehouse talent is being felt across businesses...

8 Reasons to Make the Switch to ELT Automation

Extraction, loading, and transformation (ELT) processes have been in existence for almost 30 years. It has been a programming skill set mandatory for those responsible for the creation of analytical environments and their maintenance because ELT automation works....

What is a Data Model?

A data model depicts a company's data organization, standardizing the relationships among data elements and their correspondence to real-world entities' properties. It facilitates the organization of data for business processes and information systems, offering tools...

Webinar Recap: Navigating the Future of Data Analytics

In an era where data is the new gold, understanding its trajectory is crucial for any forward-thinking organization. Our recent webinar, "Capitalizing on Data Analytic Predictions by Focusing on Cross-Functional Value of Automation and Modernization," hosted in...

Introducing: Data Automation Levels

The concept of automation has seamlessly integrated into many aspects of our lives, from self-driving cars to sophisticated software systems. Recently, Mercedes-Benz announced their achievement in reaching Level 3 in automated driving technology, which got me thinking...

Agile Data Warehouse Design for Rapid Prototyping

Agile Prototyping: Revolutionizing Data Warehouse Design While most people know WhereScape for its automated code generator that eradicates repetitive hand-coding tasks, there is another major way in which the software can save huge amounts of time and resources....

Related Content

How to Hire and Retain Data Warehouse Developers

How to Hire and Retain Data Warehouse Developers

The projected data warehouse developer job growth rate is 21% from 2018-2028, with about 284,100 new jobs for data warehouse developers projected over the next decade, according to Zippia. This surge in demand for data warehouse talent is being felt across businesses...

8 Reasons to Make the Switch to ELT Automation

8 Reasons to Make the Switch to ELT Automation

Extraction, loading, and transformation (ELT) processes have been in existence for almost 30 years. It has been a programming skill set mandatory for those responsible for the creation of analytical environments and their maintenance because ELT automation works....

How to Hire and Retain Data Warehouse Developers

How to Hire and Retain Data Warehouse Developers

The projected data warehouse developer job growth rate is 21% from 2018-2028, with about 284,100 new jobs for data warehouse developers projected over the next decade, according to Zippia. This surge in demand for data warehouse talent is being felt across businesses...

8 Reasons to Make the Switch to ELT Automation

8 Reasons to Make the Switch to ELT Automation

Extraction, loading, and transformation (ELT) processes have been in existence for almost 30 years. It has been a programming skill set mandatory for those responsible for the creation of analytical environments and their maintenance because ELT automation works....