Data Vault 2.0
Data Vault 2.0 is a database modeling method published in 2013. It was designed to overcome many of the shortcomings of data warehouses created using relational modeling (3NF) or star schemas (dimensional modeling). Speci fically, it was designed to be scalable and to handle very large amounts of data. When it was released, a “large” data warehouse may have hundreds of gigabytes or even a terabyte or two. Furthermore, as warehouses become more complex, as more source systems were introduced, the shortcomings of 3NF and star schemas were even more obvious.
Data Vault Scalability
One of the primary design goals of Data Vault 2.0 was to design the data warehouse to be extremely scalable. This is done through two primary characteristics of a data vault. First, the records can be inserted in parallel. With a parent-child relationship, such as order headers and details, the headers usually have to be inserted before the details. Some database engines will not allow you to insert “orphan” records until the parent is loaded. Because of the way Data Vault creates and uses hash keys, every record type can be inserted independently of any other record type. That allows for total parallelism during the load process.
Second, the data vault is designed as a “insert-only” model. With no updates or deletes of records, there will be no transactions or locks on the database during the load process. In addition, there is no database-level referential integrity, so no need to read through primary keys to ensure that relationships are maintained.
Data Vault 2.0 Methodology
While data warehouses used to be what we would now call a data mart, growth of data, both internally- and externally-generated, along with using more diverse systems as sources, having a data warehouse model that supports quick and easy modification and addition of source systems is critical. The Data Vault 2.0 methodology does this. Adding new source systems, or modifying your model as source systems change, is also handled on an “insert-only” basis. Existing tables and models are not changed, rather new tables and relationships are created. This greatly reduces the scope of testing, as well as reducing the chances of introducing errors.
Data Vault 2.0 Considerations
There are other reasons why Data Vault 2.0 may be the right model for you, such as traceability, working closer with the business, and greater auditability. From a non-technical perspective, these can be as important as the scalability and adaptability of Data Vault 2.0.
Is Data Vault 2.0 Still Relevant?
The Data Vault 2.0 design was released in 2013. It has had several updates now, but the question certainly can be asked: “Is it still relevant?” or “Is there something newer that is better?”
The two greatest factors in making data warehouses has been the exponential (literally) growth of data and source systems. Mergers and acquisitions dictate that disparate systems be integrated into a whole. New data formats (Restful APIs, JSON files, streams, IoT) have all made integration of new data sources critical to the relevancy and lifespan of a data vault. Data Vault 2.0 meets those challenges head-on. For more on WhereScape’s disparate data systems solutions, view this case study.
Data Vault 2.0 is a mature and tested methodology. It has certainly met the challenges of the past and present, and is well positioned to continue to be the major data warehouse design methodology in the foreseeable future. For an enterprise data warehouse, there is no other architecture out there right now that meets the needs of today.
Data Warehouse Automation
Since the initial release of Data Vault 2.0 in 2013, Dan Linstedt, its creator, has said that the single key to success in the design, development, and operation of a data vault has been automation. Warehouse automation software, whether it is for data vault or another methodology, consistently increases productivity, reduces errors, and helps you create a better final product for your data analysts and for your organization. In addition, a data warehouse automation tool, such as WhereScape, allows you to migrate to new targets in the future, whether it is Snowflake, Databricks, Microsoft, or nearly any other data warehouse platform. .
You can check out the many benefits of data warehouse automation for data vault here.
