Data Warehousing Automation

There’s a syndrome that analysts have observed: let’s call it the Success Conundrum. It has two defining features that you’ve undoubtedly experienced: 

  1. The more you give people, the more they want: 

Demand, where data-driven decision-making is concerned, is not only perennial, it is insatiable. 

  1. The decision support infrastructure is perceived as outdated as soon as its introduced: No matter how sophisticated, comprehensive or useful the current data warehouse is, users perceive it as not meeting their newest requirements. Decision makers want what they don’t yet have and that becomes their focus. 

The problem of rising user expectations has always been with us. The difference is that the gap between what users expect and what IT organizations are resourced and funded to provide has widened over time. The relatively well-funded and autonomous IT organizations of the 1990s were, by 2000 or so, being told by senior management that the secret to IT success lay in learning how to do more with less: lowered budgets for entitlement, new application spending controlled by the business, and reduced IT headcount. 

At the very moment when most IT organizations were struggling mightily to maintain hard-won gains in business intelligence with fewer people and smaller budgets, the Big Data revolution both diminishes those hard-won gains and creates an entirely new class of demand for data and applications within business and functional groups. 

Automate ETL

To break out of the Success Conundrum, an IT organization has to change its mindset about the process of producing a decision support infrastructure — to move from a model that emphasizes the slow and expensive process of artisan craftsmanship, to a model that focuses on acceleration and automation, replacing human labor with software to reduce cycle time and cost. 

In other words, the answer to the dilemma many IT organizations find themselves in is not “Give me more money, more people and more time, and I’ll do what I have done in the past.” The answer to the dilemma is: change how we produce decision support infrastructure to work effectively in a time when we have fewer people, less money and less time to produce more than what we were expected to produce in the past. This change — from artistry to acceleration and automation — is easy to understand, and has obvious benefits for IT leadership. 

Data Warehouse Automation 

IT leaders who have made the change successfully agree that the secret is to place the emphasis for the transformation on ingenuity. Data warehouse automation frees trapped ingenuity, and liberates an IT organization to pursue strategies for differentiating the company through data-driven decision-making. It’s a strategy for shifting resources —human and capital resources — away from repetitive tasks to value creation, and restoring data warehousing teams to relevance and leadership in their organizations. 

Data warehouse automation is an integrated platform of tools designed to automate routine IT tasks associated with designing, building, operating and modifying data warehouses and data marts, or 

accelerating those tasks that cannot be completely automated. Implementing data warehouse automation methods and tools allow IT teams to: 

  • Respond in days to business requests with accurate time, cost, and resource estimates 
  • Deliver completed data warehouses, data marts, and BI environments in far less time 
  • Rework existing data warehouses, data marts, and BI environments in response to business changes in hours or days 


The key to a successful transition from artisan-style data warehousing to data warehouse automation is choosing your initial project carefully and then increasing the usage consistently over time. 

The first projects you choose as targets for data warehouse automation and acceleration should be projects about which your internal customers 

care a great deal — projects they believe are critical to their success, in which your ability to deliver value rapidly is particularly important. Those initial projects can be a ground-up build of a new data warehouse, a migration of an existing data warehouse to a new technology infrastructure, construction of a data mart, consolidation of multiple data marts, or data warehouse augmentation. 

In your data warehousing, business intelligence and big data teams, automation has the potential to free scarce resources for high-value tasks, and to allow IT teams to rebuild cordial, collegial relationships with internal customers in business units and functional groups — customers whose attentions, and budgets, are perhaps shifting toward other sources of ingenuity for their business intelligence and advanced analytical needs. The stakes are high.