WhereScape 3D: How to handle model conversion rules

| December 17, 2020
WhereScape 3D: How to handle model conversion rules

This blog was written by Marcel Schwarze, Consultant at b.telligent, one of our trusted DACH implementation partners. Click here to read the original blog post in German and here to go to b.telligent’s partner page on our website.

Model conversion rules in WhereScape 3D are used to automatically generate data models and customize them according to individual requirements. Due to the high level of standardization and automation capacity required, model conversion rules are particularly useful for Data Vault models.  

With growing demands from the business, customization and development of model conversion rules can become more complex and complicated. This blog provides you with some best practices, tips and tricks to handle your model conversion rules successfully and simplifies their usage for beginners.

Tip #1: Look for recurring patterns (assessment of necessity)

An essential precondition for automating with model conversion rules is the availability of repeatable patterns. Process execution with model conversion rules only makes sense if certain patterns and regularities are in place for various process steps. Only then can the advantages of automation can be fully realized.

If you have a very individual process, i.e. the process is either executed only once or each new process execution significantly differs from the previous execution, then there aren’t any patterns and regularities available and it is difficult to standardize the process. In this case, the effort to implement a process with model conversion rules exceeds the actual benefit of automation and we recommend a manual implementation.

Tip #2: Implement model conversion manually before automating

Before starting to develop a new model conversion set, i.e. the automation of a process, we recommend that you execute your required process steps and have the process properly documentation first.

This has three key benefits:

  1. Assessment of feasibility: You can only automate the desired requirements if they are manually feasible. This prior manual execution will probably save you time as you don’t have to work on an automated, in most cases more time-consuming solution first.
  2. Simplification of rule selection: Due to the prior documentation of your process execution, you gain an overview of the process steps you need to map with model conversion rules. Even though not every process step equals a certain rule type, it facilitates your selection of required rule types in terms of type and extends them significantly. In this context, consider that for one process step you need multiple rule types and that multiple process steps can be covered by one rule type.
  3. Optimization of rule development: Gaining knowledge about your individual process steps, the relations and mutual interactions, will ultimately simplify your development (templates, matching criteria, …). 

Tip #3: Simple but useful: use the help function

During your rule development, the best support is provided by WhereScape’s help function. Check out the navigation path Help > Help. The sections named “Model Conversions” and “Pebble Templates” offer detailed descriptions of individual rule types and syntax of the Pebble template language.

Tip #4: Use the preview function

in WhereScape 3D you can see the result of your template code application during development and therefore reduce the development time and error rate of your implemented template snippets.

You will find the preview function to the right of the code editor. To use it, go to “Choose entity to preview template”, click on the “…” button and choose an entity. Then the Pebble code from the preview window to the left is applied to the selected entity and the result is displayed on the right-hand side.

Special tip:

By activating the auto-refresh, each change on the left-hand screen will automatically refresh the preview window. Without auto-refresh you have to click on the refresh symbol to refresh the preview window.

Tip #5: Debug your model conversions by using the snapshot functionality

If the execution of a model conversion set fails, an error message appears. This gives you information about failed rules, the object on which the rule has been applied in case of failure, and an error description.

If the information for debugging is not sufficient or a successful but incorrect model has been created, we recommend you debug by using the snapshot functionality: A snapshot during rule execution is like a snapshot of a model after a certain rule has been applied to this model. Especially before an incorrect rule (“How does a model look like before something went wrong?”) and for the incorrect rule itself (“How does a model look like after something went wrong?”) the use of snapshots is reasonable.

You can create a snapshot in just two steps:

  1. The first step is to select the rules after whose successful execution a snapshot should be created. Navigate to the model conversion rules, click on the left-hand field of one or multiple rules, and select “Snapshot”.
  2. In the second step, the corresponding rules must be executed. Here you have to ensure that the field “Create snapshots” is activated during the selection of the model conversion sets to be executed.

After running through the rules you can view the models created based on the snapshots and identify any aspects that may have been implemented incorrectly.

Tip #6: Define clear development rules

To ensure a unified development and obtain consistent and clearly structured model conversion sets, we recommend defining development guidelines. The following considerations will help:

  1. Creation of new versus extending existing model conversion sets: For clarity, you should avoid creating a new model conversion set for each new rule. Conversely, not all rules can be combined into one single model conversion set. In fact, for each new development, you should evaluate whether creating a new or extending an existing model conversion set makes more sense. Ask yourself the following questions: Can you assign a new rule to an already existing range of tasks or should it have its own topic area? Does the complexity of all new rules justify a different model conversion set?
  2. Prevention of negative interactions: Often several users develop in parallel. To avoid negative interactions due to simultaneous modifications of existing rules you should work with duplicates and transfer them after successful testing into the original model conversion sets.
  3. Compliance of naming convention: Finally, create a naming convention to ensure you keep a clear overview. More on this in the next section. 

Tip #7: Create and use a naming convention

To ensure consistency and standardization as well as to simplify the search for developed rules, we recommend creating and using a naming convention. Including the following name components in the name of a rule set applied during the transition from model category A to model category B has proven effective:

[IndividualToken]_[AbbreviationModelcategoryA][AbbreviationModelcategoryB] – [Short description of rule sets]

[IndividualToken] corresponds to an in-house abbreviation. This is to distinguish from the default rules supplied by WhereScape.  [AbbreviationModelcategoryA] and [AbbreviationModelcategoryB] are the categories of the initial model as well as the newly generated model.

As an example, let’s look at rule set “ws3d_rvls – Create loads”: This is a rule set (ws3d) provided by WhereScape which is applied during the transition from Data Vault category (RV=RawVault) to load and staging category (ls) and is used to generate load tables (Create loads).

For the rules within a rule set, we recommend you use meaningful wording and maintain the description fields properly. Additionally, you can define semantic sections via rule type “Separator” within the model conversion set.

Tip #8: Regularly export existing moel conversion sets

To reduce the risk of an accidental loss and to import legacy development stages, we recommend you regularly export existing model conversion sets in the form of XML files.

To do so, navigate to the model conversion rules and click on the displayed disk icon. Afterward, you can select the Model Conversion Sets of the current repository to be exported. This means that an entire model conversion set is exported instead of only a single rule. Use the arrow next to the disk to import model conversion sets.

Alternatively, the entire repository can be backed up automatically using CLI calls and the WhereScape RED scheduler. The import of individual model conversion sets is no longer possible. Instead, the entire repository backup must always be imported.

Special tip:

Usually, in 3D you would run different rulesets one after another. If you want to combine rulesets you can export them and copy the rules of one XML data file into the other. You can then import these results into 3D. 

Tip #9: Introduce external model conversion documentation

Often several people work on current model conversion rules at the same time. As well as inserting new rules or deleting old ones, the modification of existing rules is a critical operation. It often involves small changes to the template code or the matching criteria that may remain undiscovered or are quickly forgotten by other team members if they are not discussed.

To avoid future confusion and guarantee efficient testing, we recommend you introduce an external model conversion documentation. The following table shows a sample structure for a model conversion set named “Generate Data Vault”. You can also use it for the documentation of other rule sets and can complement backups if necessary.

In this table, the rule “Create satellites” is changed and later deleted, whereas the rule “Create hubs” is recreated.

For large DWH projects with individual requirements, we recommend sticking to certain standards to guarantee efficient and effective use of model conversion rules. In this blog article we showed you sample tips and tricks to help you get started easier.

This blog was written by Marcel Schwarze, Consultant at b.telligent, one of our trusted DACH implementation partners. Click here to read the original blog post in German and here to go to b.telligent’s partner page on our website.

How to Hire and Retain Data Warehouse Developers

The projected data warehouse developer job growth rate is 21% from 2018-2028, with about 284,100 new jobs for data warehouse developers projected over the next decade, according to Zippia. This surge in demand for data warehouse talent is being felt across businesses...

8 Reasons to Make the Switch to ELT Automation

Extraction, loading, and transformation (ELT) processes have been in existence for almost 30 years. It has been a programming skill set mandatory for those responsible for the creation of analytical environments and their maintenance because ELT automation works....

What is a Data Model?

A data model depicts a company's data organization, standardizing the relationships among data elements and their correspondence to real-world entities' properties. It facilitates the organization of data for business processes and information systems, offering tools...

Webinar Recap: Navigating the Future of Data Analytics

In an era where data is the new gold, understanding its trajectory is crucial for any forward-thinking organization. Our recent webinar, "Capitalizing on Data Analytic Predictions by Focusing on Cross-Functional Value of Automation and Modernization," hosted in...

Introducing: Data Automation Levels

The concept of automation has seamlessly integrated into many aspects of our lives, from self-driving cars to sophisticated software systems. Recently, Mercedes-Benz announced their achievement in reaching Level 3 in automated driving technology, which got me thinking...

Agile Data Warehouse Design for Rapid Prototyping

Agile Prototyping: Revolutionizing Data Warehouse Design While most people know WhereScape for its automated code generator that eradicates repetitive hand-coding tasks, there is another major way in which the software can save huge amounts of time and resources....

Related Content

How to Hire and Retain Data Warehouse Developers

How to Hire and Retain Data Warehouse Developers

The projected data warehouse developer job growth rate is 21% from 2018-2028, with about 284,100 new jobs for data warehouse developers projected over the next decade, according to Zippia. This surge in demand for data warehouse talent is being felt across businesses...

8 Reasons to Make the Switch to ELT Automation

8 Reasons to Make the Switch to ELT Automation

Extraction, loading, and transformation (ELT) processes have been in existence for almost 30 years. It has been a programming skill set mandatory for those responsible for the creation of analytical environments and their maintenance because ELT automation works....

How to Hire and Retain Data Warehouse Developers

How to Hire and Retain Data Warehouse Developers

The projected data warehouse developer job growth rate is 21% from 2018-2028, with about 284,100 new jobs for data warehouse developers projected over the next decade, according to Zippia. This surge in demand for data warehouse talent is being felt across businesses...

8 Reasons to Make the Switch to ELT Automation

8 Reasons to Make the Switch to ELT Automation

Extraction, loading, and transformation (ELT) processes have been in existence for almost 30 years. It has been a programming skill set mandatory for those responsible for the creation of analytical environments and their maintenance because ELT automation works....

What is a Data Model?

What is a Data Model?

A data model depicts a company's data organization, standardizing the relationships among data elements and their correspondence to real-world entities' properties. It facilitates the organization of data for business processes and information systems, offering tools...