We applaud the ETL vendors - they have finally recognized that databases such as Teradata are actually pretty good at processing data. Unfortunately the ETL vendors are still missing a critical point. Source systems are relatively constant - how often does an airline change its reservation system or a bank change its core banking platform? - but user requirements are not. Rapid and discontinuous business growth absolutely mandates a flexible data warehouse that can be rolled out fast and easily enhanced. As you move away from the source systems (the sweet spot of extract, transform and load tools) and into the data warehouse, especially the presentation layer, the rules change. Users want functionality now and changes made this afternoon. Keeping the data warehouse relevant is all about speed of delivery and the ability to rapidly make change. Heavy lifting tools are less relevant in this world. The difference is the same as interstate highways and inner city roads. Interstates are (relatively) stable and (hopefully) well planned. Repairs and maintenance are carried out after hours by teams of professionals to minimize disruptions. Conversely inner city roads seem to be in a constant state of upheaval. They are always being dug up, traffic is held up, and delays constant. A different set of tools are required to handle this volatility. As data moves closer to the user a rapid development becomes more important, and the capabilities of data movement tools less relevant. So, while ETL tool vendors have acknowledged that processing in the database is the right thing to do, they are still no closer to addressing the problems solved by rapid data warehouse development environments.
Interstates and inner city roads
- 01 November 2007