Published in DM Review in March 2002.|
Printed from DMReview.com
Data Warehousing Lessons Learned: Planning a Data Warehouse? Assemble the Cross-Functional Team!by Lou Agosta
Giga surveys over the past 18 months have indicated that more than 65 percent of Giga clients have data warehousing systems in production. This clearly indicates the availability of a critical mass of experience as to what works and what doesn't. Giga recently asked end-user clients what they consider the weaknesses in their production data warehouses as well as what they would do differently if they had the chance. The results represent an opportunity to avoid the mistakes of others. Attention to these lessons learned is critical to reducing the risk of surprises. This, in turn, allows the data warehousing system to become a focal point for organizations building high-impact business applications. Such applications typically include those that substitute information for inventory (supply chain management), customer relationship management (CRM) and market trend analysis.
In the most recent Giga survey of data warehousing (completed late 2001), end-user clients were asked what they consider the weaknesses in their production data warehouses as well as what they would do differently if they had the chance. One key finding: Surprises such as technical and organizational misalignment, solving the wrong problem, disappointed customers, or scheduling or budget overruns were often found and caused serious problems. The overriding recommendation is that in order to build and operate the data warehouse effectively, you must assemble and manage a cross-functional team with strong executive support and a clear business case.
One perceptive respondent observed, "Our data warehousing project was not driven by strong, visible business requirements and had no high visibility sponsorship. It died. In addition, the chosen domain was very complex, the overall architecture was far from optimal as many tables were not even normalized, and the technology used was not mature."
Had a data warehousing readiness assessment been completed beforehand, this project would never have moved beyond the planning stages. Much wasted effort would have been saved. Before undertaking a second try, a readiness assessment is essential. Take a step back, find a sponsor, agree on business requirements, obtain agreement that the sponsored goals will be supported in measurable and visible ways, roll up your sleeves and see what you can salvage. The second try should focus on a priority business process.
The readiness assessment should include a high-level inventory of information assets and an assembly of key players in the cross-functional team (as mentioned earlier) including the sponsor. Almost every business process has software and an IT system as part of its implementation. The IT department often addresses the business process from the bottom up and therefore fails to appreciate that the data warehouse is an enabler rather than an end in itself. This results in the unsuccessful data warehouse in a vacuum presented in this client vignette. Because data warehousing is a mature technology that got its start as a separate specialty in the early 1990s, there is no excuse for selecting a technology that "was not mature." The IT function bears full responsibility for this and can make a substantial contribution by doing its homework about design and technology options and acting as the honest broker to propose an architecture and infrastructure that is proven and with which it has experience.
As if further evidence were needed, another survey respondent states, "The major challenges are not technical. The problem is aligning development and deployment to an evolving organizational model and adapting to the cultural change in the move from local to global information systems."
In order to provide proper incentive for promoting the values of the enterprise as a whole, executives who perform staff evaluations (reviews) must be prepared to solicit input from the cross-functional data warehousing team (and do so with the understanding of those being evaluated).
The data warehouse can become a focal point for organizational dynamics, drawing groups out of their internal, functional silos, enabling and requiring them to grasp the enterprise perspective. However, the journey to a data warehouse may proceed slowly and be a process that draws staff out of their comfort zones. The imbalance between the need for change and the ability to change is a challenge faced by many organizations. Transitioning from local to global information systems, as described in this client vignette, can result in pushback from the local staff that is surrendering some autonomy in favor of increased overall coordination at the enterprise level. Management must make the case that benefits will outweigh the costs of undertaking the cultural transition. The reasons most commonly given for the need are the need for better resource utilization, lower IT costs, better project execution and the elimination of silos organized around systems and internal customers. Obviously, overcoming silos is critical in aligning the data warehouse with the organizational dynamics. The one best method of facilitating that is using and leveraging the cross-functional team.
Lou Agosta is the lead industry analyst at Forrester Research, Inc. in data warehousing, data quality and predictive analytics (data mining), and the author of The Essential Guide to Data Warehousing (Prentice Hall PTR, 2000). Please send comments or questions to email@example.com.
Copyright 2005, Thomson Media and DM Review.