I have two claims to fame in the technology world. The first is that I once went on a date with Julia Packard. The second is that I worked on two projects with data warehouse guru Ralph Kimball back in the early 90s, just after his first book was published. So I was excited to learn he was going to be speaking at the TDWI chapter meeting here in San Francisco. WhereScape is a sponsor of this TDWI chapter and all of their events have been great, but this one promised to be extra special.
It was interesting to hear Ralph speak and think about what has changed in our industry over 15 years. One thing was immediately obvious: I've lost a lot of hair. I had a pony tail back in '92 which has now been replaced by a vast shining dome. Ralph was already balding back them and from what I could see now his hairline has neither advanced nor receded.
More to the point, Ralph spoke about the advancing tidal wave of data that we are all facing. "I thought the data warehouse problem was solved in 1997," he said. "Between banks, retailers, and insurance companies it seemed like we had captured all of the data that humans could generate." Deep breath. "Boy, was I wrong." He described how the web generates levels of data unanticipated by anyone. Some companies are generating as much as 10 terabytes PER DAY. "Any Teradata employees out there?" he said to the TDWI crowd. "I think it's time to change the name of your company, though I don't know what you'll change it to that won't be obsolete in a few months. How about just 'Lotadata'."
Another issue in DW today is data frequency. "Some departments are demanding their data in 'real time', even though they don't know what that means. I'll give you the precise definition of 'real time'," Ralph said. "Real time' just means faster than your ETL can deliver it." This brought out a lot of knowing laughter. "No matter how fast your ETL is, someone will want it faster." There was discussion of new database technologies, but when it comes to delivering real information, speed is not necessarily the highest priority. "Even if you have an infinitely fast database appliance," he said, "it's worthless if it gives incorrect answers."
He also talked about the business pressures we are facing: the enterprise need to get more out of the data but spend less. "So given this environment, how can we do to make a difference in 2009?" he asked. "What project should we take on?"
What came next could have been right out of the WhereScape RED sales presentation. He is a strong advocate of agile methods for DW: "Get rid of the traditional software development life cycle," he said. "You want a short development cycle that is driven by users. The users should be running the project and you should be putting
something new in front of them every 20 days."
He advised carefully targeting your 2009 projects: "Work with only one or two fact tables at a time," he said. "Start with only one or two dimension tables. If you get those right, your users will have plenty to do as you move forward. Don't waste time and money on subject areas that aren't going to help the business right away."
Then, Ralph being Ralph, the presentation took off into outer space. He talked about Service Oriented Architectures and Business Process Execution Language and other stuff that I couldn't follow. I could feel my eyes glazing over and a sudden pressing need for a Starbuck's Venti no foam soy milk triple latte. But I forced myself to pay attention. Ralph, after all, has a track record of being twenty years ahead of his time.
After the presentation I went up to shake his hand. I asked him if he remembered the Monsanto project that we worked on together. "Sure," he said. He squinted and smirked. "I see you got rid of the pony tail," he said.