DM Review Published in DM Review in September 1999.
Printed from

Meta Data and Data Administration: Meta Data ROI: The Evolution of Technology, Part 2

by David Marco

In order to fully understand meta data's value, it's imperative to identify the changes impacting businesses today.

This month's column on meta data return on investment (ROI) examines the technical challenges that arise as a result of the evolutionary corporate changes.

The days of building an operational system and using it for 10 to 20 years without major enhancements (typically complete rewrites) are over. In order for companies to adapt to the market's rapidly changing landscape, the enterprise's data processing systems must be more flexible and robust than ever before. Many of our current systems have been built as "stovepipe" applications, meaning that they do not communicate easily with other corporate systems. Moreover, these stovepipe applications form their own system "islands" with their own hardware platforms, databases and development languages. Corporations are demanding new systems changes at an astounding rate; and, unfortunately, these old legacy systems do not adapt well to change.

This is most clearly evident with the Y2K problem that has haunted most every company in the world. If we stop and think about it, Y2K is merely a date field that has been designed to hold a two-digit year, instead of a four-digit year. When the problem is considered in these terms, it does not appear to be very significant. Of course, we know that the Y2K problem is very significant because the code that manipulates the date field is very convoluted, difficult to identify and hard to locate.

For those who believe that the need to make global systems changes end with Y2K, keep in mind that immediately following the Y2K will be the new standardized European currency (EURO) challenge. While the EURO will significantly impact all corporations, financial institutions will be most affected.

Along with the rapid change in business needs, there is an equally rapid change in the technology that fuels systems development. As a result, corporations are working feverishly to keep their systems up to speed with the extreme rate of change in technology. It seems that every week there is a new software application or piece of hardware that would aid a company in meeting their corporate goals. This situation is certainly not going to slow down in the upcoming years. Corporations are demanding faster hardware and more software functionality and integration to gain a corporate advantage.

Another challenge facing companies today is getting their systems to meet the needs of their business users in an understandable manner and language. As an industry, we are still not sufficiently meeting the needs of our business users. One recent survey asked CEOs if their IT systems were meeting the needs of their businesses. Over 84 percent of the CEOs felt that their systems do not. As an IT professional, this is a statistic that cannot go unnoticed. Instead of designing systems that speak in business terms, the systems we are building still communicate in IT terms. For example, I experienced a system that showed the user a report called XC001AB, which is a report that shows product sales by region and by marketing campaign over time. Clearly, a marketing analyst would much rather see this report entitled "Product Sales by Region and by Marketing Campaign Over Time." This small example illustrates why so few senior executives believe that their corporation's IT strategy is well integrated with their business strategy.

The good news is that the vast majority of these senior executives do believe that their systems are critical to the success of their business. Operational systems are typically designed to manage products. These systems have evolved over the years to produce, deliver and invoice products and services. Unfortunately, when remedial questions are posed Who are our most profitable customers? Which segment of our market offers the greatest future potential profit? Which of our products are complementary (market basket analysis)? Which competitors pose the greatest threat to our existence? Which product/service is providing the greatest value to our customers? the operational systems cannot easily or quickly provide answers to these fundamental questions that are redefining the world of business.

The emergence of decision support looks to transition legacy (operational) systems to decision support systems (DSSs) that help companies answer these questions which support a one-to-one marketing future. The DSS system is designed to manage customers, not products. DSSs are built to handle the "strategic" questions that a company's key decision-makers need to answer. Moreover, companies now understand that without a DSS, they will not be able to compete effectively enough to survive in tomorrow's marketplace.

It is easy to see the need for decision support systems. Unfortunately, the task of implementing these systems is anything but easy. The challenges for implementing a DSS system (also known as a data warehouse) come from both the business and technical arenas.

The most common cause for DSS project failure is that once the system is built, it does not meet the business objectives of the organization. Data warehouses that do not satisfy the business users' decision support needs are not accessed, and they eventually die.

Business objectives can be clearly defined and are measurable. This activity is critical since once the DSS project is completed, the management team will have to justify the expenditures. Moreover, it's important to understand that a data warehouse is not a project; it is a process. Data warehouses are organic in nature. They grow very fast and in directions that are very difficult to anticipate. Most warehouses double in size and in the number of users in their first year of production. Once a cost justification can be quantified for the initial release, the process for gaining funding for the follow-up releases is greatly simplified.

From a political perspective, an enterprise data warehouse requires consent and commitment from all of the key departments within a corporation. This challenge is difficult since it is rare that separate departments agree. As a result, the political aspects of implementing these systems are challenging.

DSS projects technically stretch an organization in ways unlike that of traditional systems projects. Typically, a data warehouse will source data from most, if not all, of the key operational systems within a company. The task of integrating this data from across the enterprise is considerable. For example, it is probable that a company would want to store customer data in the data warehouse. More than likely, there is customer data in several of the firm's operational systems. All of this dispersed customer data has to be integrated and cleansed before it can be loaded into the data warehouse a complicated process.

DSSs tend to store large quantities of data. It is even becoming somewhat common to see one or more terabytes of data stored and accessed. By adding massive amounts of data into the equation, the points of failure increase significantly. Moreover, large volumes of data push the envelope of the database management system (DBMS), middleware and hardware. If massively parallel processing (MPP) architecture is needed, parallel development techniques must be used. Keep in mind, the answer to many of these challenges comes in the form of a hefty price tag. As a result, adding the dimension of size can be painful and costly.

Many legacy systems contain redundant or inaccurate data. This lack of data quality in the operational systems has caused more than one decision support effort to fail. As a result, operational data must be cleansed before it is loaded into the data warehouse to ensure its usability. Meta data is critical for monitoring and improving the quality of the data coming from the operational systems. Meta data tracks the number of errors that occur during each data warehouse load run and can report when certain error thresholds are reached. In addition, the DSS data quality metrics should be stored over the history of the DSS. This allows corporations to monitor their data quality over time.

Next month's column will wrap up this series on meta data ROI by examining the solutions meta data provides to these business and technical challenges.

David Marco is an internationally recognized expert in the fields of enterprise architecture, data warehousing and business intelligence and is the world's foremost authority on meta data. He is the author of Universal Meta Data Models (Wiley, 2004) and Building and Managing the Meta Data Repository: A Full Life-Cycle Guide (Wiley, 2000). Marco has taught at the University of Chicago and DePaul University, and in 2004 he was selected to the prestigious Crain's Chicago Business "Top 40 Under 40."  He is the founder and president of Enterprise Warehousing Solutions, Inc., a GSA schedule and Chicago-headquartered strategic partner and systems integrator dedicated to providing companies and large government agencies with best-in-class business intelligence solutions using data warehousing and meta data repository technologies. He may be reached at (866) EWS-1100 or via e-mail at

Copyright 2005, SourceMedia and DM Review.