Portals eNewsletters Web Seminars dataWarehouse.com DM Review Magazine
DM Review | Covering Business Intelligence, Integration & Analytics
   Covering Business Intelligence, Integration & Analytics Advanced Search

View all Portals

Scheduled Events

White Paper Library
Research Papers

View Job Listings
Post a job


DM Review Home
Current Magazine Issue
Magazine Archives
Online Columnists
Ask the Experts
Industry News
Search DM Review

Buyer's Guide
Industry Events Calendar
Monthly Product Guides
Software Demo Lab
Vendor Listings

About Us
Press Releases
Advertising/Media Kit
Magazine Subscriptions
Editorial Calendar
Contact Us
Customer Service

Data Integration Adviser:
The Data Shadow System Conundrum

  Column published in DM Review Magazine
May 2006 Issue
  By Rick Sherman

As Martha Stewart would say, getting information into the hands of business users is a "good thing." Every enterprise needs to provide accurate, timely information for business groups to fuel the operational decisions they make each day.

So does it matter how they get this information? Business users may wonder what difference it makes as long as they get the information they need every day to keep their business selling products and services, working with suppliers and partners, and competing in the marketplace.

Well, it does matter how business users get information. It matters because, quite often, they get it from data shadow systems, which are groups of spreadsheets and local, customized databases - often Microsoft Access and statistical databases - created by business groups to gather data for their users. While these systems provide exactly the information that business users are asking for, they are rarely part of an enterprise's official data warehouse corporate performance management strategy. Outside the purview of the IT group, they often spawn data silos with the usual problems of inconsistency and quality.

How to Spot a Data Shadow System

Picture this typical scenario: the finance group is cranking out reports to examine business performance. They're looking at revenue and expenses versus budget and forecast. They're figuring out what is selling, who is selling it, how much it costs to produce and deliver, and how much profit they're making. And they are doing all of this in spreadsheets - dozens, maybe hundreds of spreadsheets.

And that's not all. They're using dozens and dozens of Microsoft Access databases to extract data from enterprise resource planning (ERP), enterprise applications and even the data warehouse to transform that data for use in their spreadsheets. Despite all the powerful business intelligence (BI) tools available, your company is using personal databases and spreadsheets for data integration!

Does this sound familiar? Not only is the finance group doing it, but marketing, sales and other organizations are also using Microsoft Access and Microsoft Excel to gather data from across your enterprise, transform it, report on it and analyze it. It is the same story at Fortune 100 companies and small to midsized firms. All the business users are doing it, including yours.

Meeting Business Needs

Business groups build data shadow systems to answer the business questions that the enterprise applications, data warehouse or reports fail to answer for them. They're filling a gap in the services they receive from their IT departments. Users may not want to get the information this way, but they don't see any alternative.

Data shadow systems satisfy the business groups' need for relevant business information - exactly what they believed the elegantly designed, vendor-built enterprise applications and IT BI systems would do for them. As data shadow systems evolve over time, they encompass more and more information, and increasing numbers of business users come to depend on them.

The Dark Side of Data Shadow Systems

Data shadow systems give business groups what they want, and everyone is happy, right? Not quite. The fact is that business users do not want to spend so much time creating these systems. Nor should they. They should be spending their time gaining a better understanding of their business, not wrestling with technology.

Because dealing with technology is not what business users do best, they cobble data shadow systems together without an overarching design. Each addition gets more difficult to implement and more costly to maintain. And when data management principles and disciplines aren't followed, data consistency and integrity suffer. Yes, data shadow systems often fulfill their business's need, but they do so in a very costly manner that uses too many resources and sacrifices data quality.

Achieving a Balance

You'd be hard-pressed to find any company, large or small, that doesn't already have at least one data shadow system. While they don't offer an optimal solution from a technology standpoint, they have many business-oriented qualities that cannot be ignored. Keeping the needs of business users in mind, it is possible to replace or rework these shadow systems with solutions that dovetail with a company's overall data warehousing architecture.

Replacement doesn't need to be a huge effort, either. In fact, make sure it doesn't turn into a multiyear IT project that implements the best-of-breed technology, standards and practices while you pay for the most expensive products in each software category. This overreliance on costly, silver bullet technology is the reason why we keep hearing about a 70 percent failure rate in data warehousing projects. It is not that data warehousing is a failed approach, but rather these mega IT projects generally don't meet expectations, nor do they provide a reasonable return on their huge investment.

Rebuilding data shadow systems is the right thing to do to ensure consistent, quality information for running a business. How one approaches this task in a way that maximizes value and ROI is a story for another day.  


For more information on related topics visit the following related portals...
Business Intelligence (BI), Databases and Strategic Intelligence.

Rick Sherman has over 20 years of business intelligence and data warehousing experience, having worked on more than 50 implementations as a director/practice leader at a Big Five firm and while managing his own firm. He is the founder of Athena IT Solutions, a Boston-based consulting firm that provides data warehouse and business intelligence consulting, training and vendor services. Sherman is a published author of over 50 articles, an industry speaker, a DM Review World Class Solution Awards judge, a data management expert at searchdatamanagement.com, and has been quoted in CFO and Business Week. Sherman can be found blogging on performance management, data warehouse and business intelligence topics at The Data Doghouse. You can reach him at rsherman@athena-solutions.com or (617) 835-0546.

View Full Issue View Full Magazine Issue
E-mail This Column E-Mail This Column
Printer Friendly Version Printer-Friendly Version
Related Content Related Content
Request Reprints Request Reprints
Site Map Terms of Use Privacy Policy
SourceMedia (c) 2006 DM Review and SourceMedia, Inc. All rights reserved.
SourceMedia is an Investcorp company.
Use, duplication, or sale of this service, or data contained herein, is strictly prohibited.