Portals eNewsletters Web Seminars dataWarehouse.com DM Review Magazine
DM Review | Covering Business Intelligence, Integration & Analytics
   Covering Business Intelligence, Integration & Analytics Advanced Search

View all Portals

Scheduled Events

White Paper Library
Research Papers

View Job Listings
Post a job


DM Review Home
Current Magazine Issue
Magazine Archives
Online Columnists
Ask the Experts
Industry News
Search DM Review

Buyer's Guide
Industry Events Calendar
Monthly Product Guides
Software Demo Lab
Vendor Listings

About Us
Press Releases
Advertising/Media Kit
Magazine Subscriptions
Editorial Calendar
Contact Us
Customer Service

The Importance of Data Integration

  Article published in DM Review Magazine
November 2003 Issue
  By David Marco

One of the key objectives of most CIOs is to integrate their organizations' data. In fact, the issue of data integration has risen all the way to the CFO and CEO levels of today's corporations. Why is data integration becoming so important to so many C-level executives?

There are several key reasons

  • Provide IT portfolio management
  • Reduce IT redundancy
  • Prevent IT application failures

Provide IT Portfolio Management

Over the years I have had the opportunity to perform more than a dozen data warehousing assessments. During these assessments, I always ask the client how much they spend annually on data warehousing. The majority of companies and government organizations cannot provide even a relatively good estimate on what they actually spend. In order to manage these and any other costly information technology (IT) initiatives, it is critical to measure each one of them. However, it is impossible to measure them when most companies do not understand them (see Figure 1). This is where IT portfolio management enters the picture.

IT portfolio management refers to the formal process for managing IT assets. IT assets include software, hardware, middleware, IT projects, internal staff, applications and external consulting. Like every newer discipline, many companies that have started their IT portfolio management efforts have not done so correctly. This article will discuss some of the keys for building successful IT portfolio management applications.

A properly managed IT portfolio allows a corporation to see which projects are proceeding well and which are lagging behind. In my experience, almost every large company has a great deal of duplicate IT effort occurring (see later section on "Reduce IT Redundancy"). This happens because the meta data is not accessible. At my company, we have a couple of large clients whose primary goal is to remove these tremendous redundancies, which translates into tremendous initial and ongoing IT costs.

Reduce IT Redundancy

CIO usually means chief information officer; however, there is another possible meaning for this acronym - career is over. One of the chief reasons for this is that most IT departments are "handcuffed" in needless IT redundancy that too few CIOs are willing to fix -- or even capable of fixing.

There are several CIO surveys that are conducted annually. These surveys ask, "What are your top concerns for the upcoming year?" Regardless of the survey, data integration will be high on the list.

Data integration has two facets to it. One is the integration of data across disparate systems for enterprise applications. The second is the integration/removal of IT redundancies. Please understand that some IT redundancy is a good thing. For example, when there is a power outage and one of your data centers is non-operational, you need to have a backup of these systems/data. However, when I talk about IT redundancies, I am addressing "needless" IT redundancy -- IT redundancy that only exists because of insufficient management of IT systems.

I was working with a Midwestern insurance company that, over a four-year span, had initiated various decision support efforts. After this four-year period, they took the time to map the flow of data from their operational systems to their data staging areas and finally to their data mart structures. What they discovered is shown in Figure 2.

What is enlightening about Figure 2 is that when I show this illustration during a client meeting or at a conference keynote address, many people ask me how I obtained a copy of their IT architecture. If you work at a Global 2000 company or any large government entity, Figure 2 represents an overly simplified version of your IT architecture. These poor architecture habits create a litany of problems including:

  • Redundant applications, processes and data
  • Needless IT rework
  • Redundant hardware and software

Redundant Applications, Processes and Data

It has been my experience working with large government agencies and Global 2000 companies that needlessly duplicate data is running rampant throughout our industry. In my experience, the typical large organization has between needless data redundancy -- three to four times more than is necessary. Moreover, I can name multiple organizations that have literally hundreds of "independent" data mart applications spread all over the company. Each one of these data marts is duplicating the extraction, transformation and load (ETL) that is typically done centrally in a data warehouse. This greatly increases the number of support staff required to maintain the data warehousing system as these tasks are the largest and most costly data warehousing activities. In addition to duplicating this process, each data mart will also copy the data -- requiring further IT resources. It is easy to see why IT budgets are straining under the weight of all of this needless redundancy.

Needless IT Rework

During the requirements-gathering portion of one of our meta data management initiatives, I had an IT project manager discuss the challenges that he is facing in analyzing one of the mission- critical legacy applications that will feed the data warehousing application that his team has been tasked to build. During our interview he stated, "This has to be the twentieth time that our organization is analyzing this system to understand the business rules around the data." This person's story is an all too common one as almost all organizations reinvent the IT wheel on every project. This situation occurs because usually separate teams will build each of the IT systems. Because they don't have a managed meta data environment (MME), these teams do not leverage the other's standards, processes, knowledge and lessons learned. This results in a great deal of rework and reanalysis.

Redundant Hardware and Software

I have discussed a great deal about the redundant application and IT work that occurs in the industry. All of this redundancy also generates a great deal of needless hardware and software redundancy. This situation forces the enterprise to retain skilled employees to support each of these technologies. In addition, a great deal of financial savings is lost, as standardization on these tools doesn't occur. Often a software, hardware or tool contract can be negotiated to provide considerable discounts for enterprise licenses, which can be phased into. These economies of scale can provide tremendous cost savings to the organization.

In addition, the hardware and software that is purchased is not used in an optimal fashion. For example, I have a client that has each individual IT project buy their own hardware. As a result, they are infamous for having a number of servers running at 25 percent capacity.

From the software perspective, the problem only gets worse. While analyzing a client site, I had asked the IT project leaders what software vendors they standardized on. They answered, "All of them!" This is reminiscent of the old joke, "What is the most popular form of software on the market? Answer: shelfware!" Shelfware is software that a company purchases but never uses; it just sits on the shelf collecting dust.

Prevent IT Applications Failure

When a corporation looks to undertake a major IT initiative, such as a customer relationship management (CRM), enterprise resource planning (ERP), data warehouse or e-commerce solution, their likelihood of project failure is between 65 and 80 percent depending on the study referenced. This is especially alarming when we consider that these same initiatives traditionally have executive management support and cost many millions of dollars. For example, I have one large client that wants to implement a global CRM system (e.g., Siebel, Oracle) and an ERP system (e.g., SAP, PeopleSoft) in the next four years. Their initial project budget is more than $125 million! In my opinion, they have no probability of delivering all of these systems on time and on budget. Consider this: when was that last time that you've seen an ERP or CRM initiative delivered on time or on budget?

When we examine the causes for these project failures, several themes become apparent. First, these projects did not address a definable and measurable business need. This is the number one reason for project failure -- data warehouse, CRM, MME or other. As IT professionals, we must always be looking to solve business problems or capture business opportunities. Second, when projects fail, it is usually because those involved have a very difficult time understanding their company's existing IT environment and business rules. This includes custom applications, vendor applications, data elements, entities, data flows, data heritage and data lineage.

MME's Focus On Data Integration

Many of these Global 2000 companies and large government organizations are targeting MME technology to assist them in identifying and removing existing application and data redundancy. Moreover, many companies are actively using their MME to identify redundant applications through analysis of the data. These same companies are starting IT application integration projects to merge these overlapping systems and to ensure that future IT applications do not proliferate needless redundancy.

If your organization can reduce applications, processes, data, software and hardware, lower the likelihood for IT project failure and speed up the IT development life cycle, then clearly it will greatly reduce IT expenditures. For example, I have a large banking client that asked my company to analyze their IT environment. During this analysis, we discovered that they have a tremendous amount of application and data redundancy. Moreover, we determined that they have more than 700 unique applications. I then compared this client to a bank that is more than twice their size. This larger bank has a world-class MME and uses it to properly manage their systems. As a result, they have less than 250 unique applications. Clearly the bank with more than 700 applications has a great deal of needless redundancy as compared to a bank that is more than twice their size and has less than 250 applications. Interestingly enough, the bank that has less than 250 applications and has a world-class MME is also 14 times more profitable than the bank maintaining more than 700 applications. It seems fairly obvious that the less profitable bank would become much more profitable if they removed this redundancy.


For more information on related topics visit the following related portals...
Data Integration.

David Marco is an internationally recognized expert in the fields of enterprise architecture, data warehousing and business intelligence and is the world's foremost authority on meta data. He is the author of Universal Meta Data Models (Wiley, 2004) and Building and Managing the Meta Data Repository: A Full Life-Cycle Guide (Wiley, 2000). Marco has taught at the University of Chicago and DePaul University, and in 2004 he was selected to the prestigious Crain's Chicago Business "Top 40 Under 40."  He is the founder and president of Enterprise Warehousing Solutions, Inc., a GSA schedule and Chicago-headquartered strategic partner and systems integrator dedicated to providing companies and large government agencies with best-in-class business intelligence solutions using data warehousing and meta data repository technologies. He may be reached at (866) EWS-1100 or via e-mail at DMarco@EWSolutions.com.

Solutions Marketplace
Provided by IndustryBrains

Data Validation Tools: FREE Trial
Protect against fraud, waste and excess marketing costs by cleaning your customer database of inaccurate, incomplete or undeliverable addresses. Add on phone check, name parsing and geo-coding as needed. FREE trial of Data Quality dev tools here.

Speed Databases 2500% - World's Fastest Storage
Faster databases support more concurrent users and handle more simultaneous transactions. Register for FREE whitepaper, Increase Application Performance With Solid State Disk. Texas Memory Systems - makers of the World's Fastest Storage

Manage Data Center from Virtually Anywhere!
Learn how SecureLinx remote IT management products can quickly and easily give you the ability to securely manage data center equipment (servers, switches, routers, telecom equipment) from anywhere, at any time... even if the network is down.

Design Databases with ER/Studio: Free Trial
ER/Studio delivers next-generation data modeling. Multiple, distinct physical models based on a single logical model give you the tools you need to manage complex database environments and critical metadata in an intuitive user interface.

Free EII Buyer's Guide
Understand EII - Trends. Tech. Apps. Calculate ROI. Download Now.

Click here to advertise in this space

View Full Issue View Full Magazine Issue
E-mail This Article E-Mail This Article
Printer Friendly Version Printer-Friendly Version
Related Content Related Content
Request Reprints Request Reprints
Site Map Terms of Use Privacy Policy
SourceMedia (c) 2006 DM Review and SourceMedia, Inc. All rights reserved.
SourceMedia is an Investcorp company.
Use, duplication, or sale of this service, or data contained herein, is strictly prohibited.