Portals eNewsletters Web Seminars dataWarehouse.com DM Review Magazine
DM Review | Covering Business Intelligence, Integration & Analytics
   Covering Business Intelligence, Integration & Analytics Advanced Search

View all Portals

Scheduled Events

White Paper Library
Research Papers

View Job Listings
Post a job


DM Review Home
Current Magazine Issue
Magazine Archives
Online Columnists
Ask the Experts
Industry News
Search DM Review

Buyer's Guide
Industry Events Calendar
Monthly Product Guides
Software Demo Lab
Vendor Listings

About Us
Press Releases
Advertising/Media Kit
Magazine Subscriptions
Editorial Calendar
Contact Us
Customer Service

Beyond the Data Warehouse:
Information Management Full Employment Acts

online columnist John Ladley     Column published in DMReview.com
January 30, 2004
  By John Ladley

During the data warehouse boom of the 1990s, there was often tongue-in-cheek talk that the data warehouse was the "Full Employment Act for database administrators (DBAs)." This was due to the explosive demand for DBAs to wring performance out of database management systems (DBMSs) that were not built for data warehouse functions.

Don't look now, but another "full employment" simile may be in the works. The reasons will have nothing to do with technology but with regulatory pressure and economic changes. I am talking about an increase in data administration and data management jobs. In addition to companies realizing that information asset management (IAM) is now a necessary part of business (see July 2003, DMReview.com article on convergence,), public companies and others are confronted with a rather complex series of legislation that are, indeed, forcing information management issues to be addressed. A savvy CIO must be proactive in light of these changes.

Examples of these legislative catalysts to data management abound. HIPAA, Graham Leach Bliley(GLB), Sarbanes- Oxley (SOX), etc. all present organizations with a legal reason to achieve elevated levels of information governance. Some of these regulations are specific (e.g., HIPAA and privacy and encryption) and some are obscure (SOX), but all are pretty clear in their intent - if an executive does not effectively manage the quality, accuracy and content of the organization's information, the organization and its executives are subject to fines and prison.

After reviewing (and, yes, reading) several of these laws, a few crucial themes are emerging. We will visit those themes later, but it is key to note that all of these themes are dependent on managing information. The consequences of failing to do so result in fines and jail time - serious considerations. The bottom line is there must be information governance. It is no longer an option. A data warehouse, Corporate Information Factory, meta data awareness, etc. represent an initial start, but there must be a lot more.

Visibility into Controls

A good data warehouse has effective controls for ensuring data integrity as do good finance systems. Rarely do the two sets of controls meet up. When they do, companies can actually use the DW for financial reporting. This is beginning to happen with greater frequency.

However, controls must now be well documented and tie to each other. (SOX Section 404). This means possible reengineering of controls for applications. Since controls are also business rules, this also means meta data needs updating. It is no longer enough to say "the data adds up."

During a recent speech, I asked a crowd of about 150 people, "Who can define for me a hash total process?" Very few hands were raised. While this was a crude survey, I submit that data administrators may have to become familiar with basic accounting controls and create meta data to support and define these. Across SOX and the other regulations, companies must prove they know what they know.

Tracking of Changes

Many regulations require tracking of changes to data. Like the controls requirements, this means meta data to track changes in dimensions and attributes as well as monitoring changes to actual data values. Adjustments to static data have been a pox on data warehouse architects. Normally, someone is allowed to "tweak" data so accuracy can be assured. This is now, potentially, an illegal act. Adjusting data must now be controlled via rules and authorizations - all meta data requirements.

Privacy and Encryption

Several of the regulations require encryption, removal of SSN as primary key and mandate clean data. HIPAA (and related state regulations, such as exist in California) mandate that SSN is never to be used as an identifier for an individual. Additionally, individuals health records cannot be identifiable by knowledge workers performing aggregate analysis.

GLB requires stringent controls on personal financial information. While seemingly a regulation that targets only banks, it affects any company that issues credit, e.g., consumer product, automobiles, builders, etc.

Again, at the risk of repetition, the management of all of this depends on meta data.

Technology Trends

All of the above implies a robust meta data layer. At minimum, some type of passive catalog that contains not only the traditional entitles and attributes, but also rules, context and, heaven forbid, documentation of the actual reports and metrics to be reported. There are no tools or repositories that explicitly offer these elements. While the top repository offerings can be extended, there are few examples and even less experience of successful use of repositories to monitor controls, context and metrics.

The technology challenge compounds when these same, aging tools (traditional repositories) must also adapt to low-latency data handling (information buses, etc.). The Internet is that data management areas will need to adopt some type of basic repository engine and then blend in some homegrown extensions.

Culture Changes

Perhaps the most significant change in store for IT will be the fundamental changes in how data administrators (DA) and DBAs execute their tasks. DAs will need to learn fundamentals of accounting controls. DBAs will need to carefully monitor access to data values. CIOs will need to pump money into both of those areas in the form of training, tools and external consultants to ensure continuity and compliance.

Training must come from areas familiar with the laws in detail. Consultants must be more than good meta data practitioners or data warehouse designer. I submit that at some point, independent auditors may need to certify IT consultants (not the lame self-promoting certifications that proliferate in our industry).

Data administration is an area that, in many companies, has been either a programmer retirement zone or a misunderstood and, therefore avoided, function. Now this group of people has the skills to keep the CxO away from jail. This is a commendable responsibility, but the CIO must confirm that the existing data management groups can handle the accountability and move to reinforce their capabilities.


In addition to architectures that manage the actual data (warehouse, CIF, etc.), there will need to be architectures that manage the meta data. There will layers of architecture that feature controls and business rules. There will be a supervisory component for compliance that can intrude into every aspect of data handling to ensure accuracy, privacy, control and whatever else regulators have in store.

Figure 1: Layers of Architectures

All of these requirements go beyond mere tools and a few procedures. Historically reliable techniques and tools may prove totally inadequate. Therefore DAs, DBAs and CIOs must be prepared to be creative, focused and accountable to their business sponsors. Meta data is no longer a nicety, but a crucial component of enterprise technology architectures.

The contents of this article are Copyright 2003 by DM Review and KI Solutions. Any use, quotation, repurpose, duplication or replication of the diagrams, concepts or content without permission of DM Review and the author is prohibited.


Check out DMReview.com's resource portals for additional related content, white papers, books and other resources.

When John is not writing poetry as a hobby, he is a director for Navigant Consulting, which recently acquired KI Solutions, a management consulting firm specializing in knowledge and information asset management and strategic business intelligence planning and delivery. Ladley is an internationally recognized speaker and, more importantly, hands-on practitioner, of information and knowledge management solutions. He can be reached at jladley@navigantconsulting.com. Comments, ideas, questions and corroborating or contradictory examples are welcomed.

E-mail This Column E-Mail This Column
Printer Friendly Version Printer-Friendly Version
Request Reprints Request Reprints
Site Map Terms of Use Privacy Policy
SourceMedia (c) 2006 DM Review and SourceMedia, Inc. All rights reserved.
SourceMedia is an Investcorp company.
Use, duplication, or sale of this service, or data contained herein, is strictly prohibited.