Published in DM Review in November 1998.|
Printed from DMReview.com
Stranded on Islands of Databy David Marco
Summary: This article addresses the characteristics of independent data marts, the flaws in their architecture and the reason they exist.There is a severe disease that has spread to epidemic proportions throughout our society. This disease is particularly dangerous as its effects are not readily identifiable at the time of infection. However, if this condition goes untreated, it can be debilitating and even terminal. This disease is not hepatitis, but rather "independent" data marts. While this imagery may seem a bit dramatic, unfortunately it reflects the reality in many of today's companies.
This article is the first of a two-part series on migrating from independent data marts to an architected solution. This installment will address the characteristics of independent data marts, the flaws in their architecture and the reasons why they exist. Part two will run in the December issue of DM Review and will address specifically how a company can migrate off of the independent data mart architecture to an architected solution.
Characteristics of Independent Data Marts
Independent data marts are characterized by several traits. First, each data mart is sourced directly from the operational systems without an enterprise data warehouse to supply the architecture necessary to sustain and grow the data marts. Second, these data marts are typically built independently from one another by autonomous teams. Typically, these teams will utilize varying tools, software, hardware and processes.
Possibly the most visually descriptive trait of a company that has constructed independent data marts is that once they map out a schema of their decision support systems (DSSs), the schema will resemble that of a "spaghetti" chart (see Figure 1).* What is most disturbing is the number of companies that have expressed that this chart resembles their current DSS architecture.
Obviously, this architecture is not an architecture at all. Instead it is a series of "stovepipe" DSS systems. This architecture greatly differs from that of an architected data warehouse (see Figure 2).
The purpose of this article is to discuss independent data marts and the process for migrating to an architected solution. However, it will briefly touch on the topic of DSS architecture. It will not go into a detailed discussion of top-down versus bottom-up approaches, except to say that the "classic" top-down approach is a more scalable and logical approach for constructing a DSS system. It is surprising how often the top-down methodology is mistaken for a "galactic" approach. This is a misunderstanding as the top-down approach is best used iteratively and incrementally to build the DSS system. When used in this fashion, the cost of building a data warehouse that feeds "dependent" data marts becomes highly comparable to the cost of building independent data marts.
Problems With Independent Data Marts
It would be enlightening if a study were conducted to calculate the costs of maintaining non-necessary redundant data for Fortune 1000 companies. The end total would be in the billions of dollars in expenses and lost opportunity.
Separate teams will typically build each of the independent data marts in isolation. As a result, these teams do not leverage the other's standards, processes, knowledge and lessons learned. This results in a great deal of rework.
These autonomous teams will commonly select different tools, software and hardware. This forces the enterprise to retain skilled employees to support each of these technologies. In addition, a great deal of financial savings is lost, as standardization on these tools doesn't occur. Often a software, hardware or tool contract can be negotiated to provide considerable discounts for enterprise licenses. These economies of scale can provide tremendous cost savings to the organization.
One of the chief phenomena facing corporations today is the current merger and acquisition craze. Interestingly enough, one of the key factors fueling this movement is these companies' desire to reduce their IT spending. In light of this situation, the costs associated with independent data marts become even more magnified as companies continue to focus on controlling their ever-growing IT costs.
It is important to note that many companies that have built independent data marts are currently in the process of migrating off of them. Needless to say, the cost--in dollars and time--for the migration is not trivial.
Why Do Independent Data Marts Exist?
With all of these architectural flaws, it would seem surprising that so many companies have built their DSS systems around this architecture. There are several reasons why this aberration has occurred.
DSSs Are Complex:
In order to construct a data warehouse, a corporation must truly come to terms with their data and the business procedures that the data represents. While this task is challenging, it is a necessary step and one from which the true value of the DSS process is derived.
Independent Data Mart Shortcut:
Inappropriate Vendor Messages:
The current vendor buzzword in today's market is "turnkey." Everyone seems to offer a "turnkey" DSS solution. Unfortunately, merely purchasing a "turnkey" solution does not alleviate the task of learning and understanding a corporation's data and their business processes. Integration of data from disparate systems requires a careful analysis and an understanding of business processes and the data that represents them. There isn't a "magic bullet" or "turnkey" solution that alleviates this task.
In the December issue of DM Review, the two approaches for migrating from independent data marts and an independent data mart migration case study will be presented.
* It is important to note that this chart is an actual client's DSS architecture schematic. I'm proud to say that they are no longer on this architecture.
David Marco is an internationally recognized expert in the fields of enterprise architecture, data warehousing and business intelligence and is the world's foremost authority on meta data. He is the author of Universal Meta Data Models (Wiley, 2004) and Building and Managing the Meta Data Repository: A Full Life-Cycle Guide (Wiley, 2000). Marco has taught at the University of Chicago and DePaul University, and in 2004 he was selected to the prestigious Crain's Chicago Business "Top 40 Under 40." He is the founder and president of Enterprise Warehousing Solutions, Inc., a GSA schedule and Chicago-headquartered strategic partner and systems integrator dedicated to providing companies and large government agencies with best-in-class business intelligence solutions using data warehousing and meta data repository technologies. He may be reached at (866) EWS-1100 or via e-mail at DMarco@EWSolutions.com.
Copyright 2005, SourceMedia and DM Review.