The Data Strategy Advisor:
Exploring the Social Life of the Data Warehouse
Business intelligence (BI) systems (especially data warehouses) are a lightning rod for organizational dynamics. Control of information is both a cause and effect of the way power is distributed in an enterprise; data warehouses distribute BI information in abundance. A successful data warehouse can create information asymmetries between the data haves and the data have-nots; people will fight to stay out of the latter group. When data warehousing systems are first installed, subjected to a service level agreement or enhanced, data warehousing politics impend. There seems to be no way to avoid this, though savvy enterprises can take steps to channel such political energies into productive pathways that make the organization smarter than the sum total of the individuals who work there. Specific steps can be taken to reduce (or manage) the risk of information hoarding, territoriality or other pathological corporate behavior reminiscent of a Dilbert cartoon.
The first step is to undertake an inquiry into organizational form - distributed, federated or centralized - and consider how the data architecture aligns or conflicts with it. Alignment of organizational form to data warehousing structure is a key determinant of project and operational success. This undercuts the religious wars between proponents of the dimensional star schema and the consolidation data warehouse (sometimes called an operational data store or ODS) that have plagued data warehousing. Data warehouse designers, implementers and operators need to be able to switch between the two in order to address tactical design issues and strategic BI initiatives in turn.
The BI consultant knows that social dynamics loom large when entering the client meeting room by observing the star schema advocates on one side of the table and proponents of the operational data store on the other side. In conversations with more than 1,000 data warehousing clients since 1999, I have often been called to mediate conflict between advocates of different data models (and data architectures) - which, after all, are different representations of the business. These disagreements cannot be solved by a benchmark or other technical decision because they are not really technology disputes. In short, the disagreements about technology often mask underlying social dynamics about who controls the information. The way out of the impasse requires a breakthrough in surfacing and aligning business and data architecture, data governance, the capacity to compromise and what used to be called the ability to play well with others.
For example, in trying to implement a large, centralized data warehouse in a highly distributed manufacturing firm, one organization found that standards and definitions recommended by the corporate center conflicted with decisions that had already been made based on the local autonomy of the distributed fabrication facilities. Turf wars broke out. Instead of a data warehouse, they built a technology tower of Babel with the same level of communication - babble. As one manager said, "Adapting to the cultural change in moving from local information systems to centralized ones required flexibility and accommodation at new levels. We weren't there." This particular organization eventually abandoned the attempt as politically impractical and focused its efforts more profitably on designing a consistent, unified representation of products, customers and suppliers to be shared by distributed analytic applications. In contrast, a centralized financial services firm allowed its local lines of business considerable autonomy in the management of proliferating data marts. Differing departments wanted the same information in slightly different forms. Redundancy was rampant. The result was excessive coordination costs as system interfaces multiplied; maintenance was constantly needed. Of course, providing the same information in slightly different forms is why the data warehouse was invented; a data mart consolidation effort finally proved successful in both reducing coordination costs and making the information more accessible.
The lesson learned is clear. No firm will reorganize its operations for the sake of a data warehouse initiative, no matter how enterprise wide. Firms with centralized organizations - airlines, telecom, finance - favor large, centralized architecture. Real-world situations result in hybrid architectures of varying complexity and coherence - federated (beverage bottlers, consumer packaged goods, retail, public sector). What is the result of going against these patterns? Social conflict: centralized firms fail with distributed architectures and vice versa. Function follows form. Aligning data warehousing architecture with organizational form means centralizing the architecture when the form of the business is centralized, decentralizing when it is distributed and federating when transitioning between the two.
The key recommendation in having a data warehouse with a well-balanced social life is to align the structure of the data warehouse to the form of the enterprise. There is no known analogy to this in getting a date for the high school homecoming dance. Still, many implementation and coordination issues need to be addressed.
The next most important action a business can take (from the point of view of representing the business in the data architecture) is to design consistent and unified definitions of product, customer, channel, etc. Master data management lies on the critical path. The team can then implement federated data marts or a centralized, atomic data warehouse or some combination of the two (such as ODS with downstream, dependent data marts). These will be critical determinants of overall technology success and operational costs. The advantage is that technical and operational issues will then be about technology and operations, not about hidden agendas to control the information. Lining up data and business architecture patterns will result in a data warehouse that will enjoy the popularity of a high school homecoming queen.
For more information on related topics visit the following related portals...
Business Intelligence (BI),
DW Design, Methodology,
Master Data Management and
Lou Agosta, Ph.D., joined IBM WorldWide Business Intelligence Solutions in August 2005 as a BI strategist focusing on competitive dynamics. He is a former industry analyst with Giga Information Group, has served as an enterprise consultant with Greenbrier & Russel and has worked in the trenches as a database administrator in prior careers. His book The Essential Guide to Data Warehousing is published by Prentice Hall. Agosta may be reached at LoAgosta@us.ibm.com.