Portals eNewsletters Web Seminars dataWarehouse.com DM Review Magazine
DM Review | Covering Business Intelligence, Integration & Analytics
   Covering Business Intelligence, Integration & Analytics Advanced Search

View all Portals

Scheduled Events

White Paper Library
Research Papers

View Job Listings
Post a job


DM Review Home
Current Magazine Issue
Magazine Archives
Online Columnists
Ask the Experts
Industry News
Search DM Review

Buyer's Guide
Industry Events Calendar
Monthly Product Guides
Software Demo Lab
Vendor Listings

About Us
Press Releases
Advertising/Media Kit
Magazine Subscriptions
Editorial Calendar
Contact Us
Customer Service

Knowledge Integrity:
Master Data Standards and Data Exchange

  Column published in DM Review Magazine
August 2005 Issue
  By David Loshin

In my previous column, we started to look at the topic of master data management and customer data integration, and how those tasks are related to each other as well as to other historical attempts at enterprise knowledge integration. As part of the integration process, this month we look at a case study involving the exchange of data using what are thought to be commonly defined data standards. This case study exposes a problem when a data standard is in place and what happens when individual participants adjust their adherence to the standard.

This topic surfaces frequently, and it relates to the use of agreed-to data set formats for data interchange whose use somehow diverges from their original intent. The changes occur subtly over time, and in potentially multiple places; therefore, until an analyst actually sits down to review the different uses, the variants would not even be noticed. Sometimes the differences are in content and sometimes they are in form, but at some point, especially when data is being exchanged, the variances will result in the inability to synchronize data values between information exchange partners.

In this case study, we were consulting with a client who sends and receives data with a number of information exchange partners. There is an agreement in place to include routing codes in exchanged messages that help redirect messages and requests to the appropriate location within an exchange partner's local environment. When partner A sends a request to partner B, A attaches a routing code related to the originator of the request within the message. B's reply to A will include the same routing code, which A then uses for redirection of the reply to the originating requestor.

Each partner is responsible for its own scheme for assigning routing codes as well as its own process for message redirection. In addition, all the partners post some version of their routing codes to a centralized repository so that anyone originating a message can look up the routing code, append it into a new message and believe that it will be routed to the proper location at the destination. However, there is some (limited) enterprise-wide guidance describing a general scheme for data formatting, and all of the exchange partners believe they conform to this guidance.

Here is where the subtle problems occur. Despite their best beliefs, a number of the exchange partners have, in their own ways, slightly modified the way that the routing codes are formed and how they are mapped internally. Those who have done this have good reasons, but they have adapted variance into the process without obtaining agreement from the other partners. There are a few results as follows:

  • Those partners that have made changes have adapted the same physical components of the code for different meanings. One partner may use a part of the code to indicate routing to a centralized office, while another uses that same part of the code (and sometimes, the same mapped values) to indicate routing to a financial accounting service.
  • Some of the partners have added extra bytes to the structure, allowing for additional information to be appended. Yet those that have not extended their data structures are unable to capture the appended values, which means that the full values cannot be incorporated in any response. When this happens, the messages are either improperly routed or are dropped altogether.

As these variations have crept into the environment, the knowledge-workers that participate in the workflows have begun to notice the existence of problems, but they have had difficulty determining the source of the problem. The reason is that even though the adjustments that each participant has made only slightly alter their conformance to the standard, in combination across the extended enterprise, a number of these variations clash with each other. As a result of the limited enterprise-wide guidance, conflicts emerge from the loose adherence to the standard. This, in turn, reduces the efficiency of the interacting systems.

For example, while partner X uses a specific code (e.g., "000") to indicate routing to a central processing office, partner Y uses the same code for routing to an EFT/EDI processor. Therefore, when partner X wants to send a message to partner Y's central processing center, X's use of the "000" code in its message results in Y routing the message to a completely unintended location.

Even though this case study looks specifically at an exchange network, the lesson learned can apply directly to data integration across an enterprise. There are always expectations of conformance to some standards, but often in reality there is no central oversight of these standards. They are rarely properly documented, and the IT staff within each line of business does not feel constrained by compliance to a standard. This complicates data aggregation and integration, especially for data warehousing and business intelligence applications.

The solution to this divergence from what is believed to be a standard must facilitate the cooperation of the participants along two dimensions: precise definition and adherence to the standard. Because of the limited guidance, there were enough loopholes in the standard to allow enough variation until conflicts became inevitable. By having more precise (or, perhaps, stricter) definitions and guidelines, the opportunity for introducing variation is reduced; and, consequently, so is the possibility of value clashes. Second, it is to each participant's benefit to completely conform to the standard. When everyone plays by the rules and knows that everyone else is playing by the rules, data expectations may be set properly. The result is a more streamlined ability to exchange and integrate information across the enterprise.   


For more information on related topics visit the following related portals...
Data Quality and Master Data Management.

David Loshin is the president of Knowledge Integrity, Inc., a consulting and development company focusing on customized information management solutions including information quality solutions consulting, information quality training and business rules solutions. Loshin is the author of Enterprise Knowledge Management - The Data Quality Approach (Morgan Kaufmann, 2001) and Business Intelligence - The Savvy Manager's Guide and is a frequent speaker on maximizing the value of information. Loshin may be reached at loshin@knowledge-integrity.com.

Solutions Marketplace
Provided by IndustryBrains

Data Quality Tools, Affordable and Accurate
Protect against fraud, waste and excess marketing costs by cleaning your customer database of inaccurate, incomplete or undeliverable addresses. Add on phone check, name parsing and geo-coding as needed. FREE trial of Data Quality dev tools here.

Design Databases with ER/Studio: Free Trial
ER/Studio delivers next-generation data modeling. Multiple, distinct physical models based on a single logical model give you the tools you need to manage complex database environments and critical metadata in an intuitive user interface.

Free EII Buyer's Guide
Understand EII - Trends. Tech. Apps. Calculate ROI. Download Now.

Click here to advertise in this space

View Full Issue View Full Magazine Issue
E-mail This Column E-Mail This Column
Printer Friendly Version Printer-Friendly Version
Related Content Related Content
Request Reprints Request Reprints
Site Map Terms of Use Privacy Policy
SourceMedia (c) 2006 DM Review and SourceMedia, Inc. All rights reserved.
SourceMedia is an Investcorp company.
Use, duplication, or sale of this service, or data contained herein, is strictly prohibited.