The Song Remains the Same

In 2003, the inaugural column I wrote discussed the Data Integration Framework (DIF). This approach is the foundation that I have used for more than 20 years to design, develop, deploy and maintain data integration solutions. Technology and products have changed, but as I’m always reminding my consulting clients, the basis for sound data integration remains. Always remember the following basics that comprise the DIF:

• Architectures that are used to stage data from data sources to information consumers;
• Processes that help gather, consolidate, transform, cleanse and aggregate data and metadata;
• Standards for ensuring consistency, accuracy, integrity and data quality;
• Resources and skills that are necessary to ensure success; and
• Tools for the creation, deployment and expansion of the framework.

Despite all the PR and sales presentations, tools are only a small piece of the framework. In fact, it is no coincidence that they are listed last on the DIF list.

A Shortcut to Nirvana

The industry loudly proclaims the arrival of an alphabet soup of technologies and products destined to provide information integration nirvana - and often at a fraction of the cost of the “old” technologies. This soup includes master data management (MDM), customer data integration (CDI), corporate performance management (CPM), service-oriented architecture (SOA), software as a service (SaaS), on-demand software (same as SaaS), open source software (OSS), data warehouses (DWs), DW appliances, data marts, online analytical processing (OLAP) cubes, in-memory cubes, enterprise information integration (EII), enterprise application integration (EAI), dashboards, scorecards, data visualization and many others.

All are being sold as the magic elixir to an enterprise’s data integration needs. They tell the same story: “This time, for sure” they will solve all your integration issues, and it will take a fraction of the time you’ve spent in the past. They remind you that your past failures are because you didn’t have this new technology. They make it sound as though the technology would have known how to define profit margins for all your enterprise business units without IT ever having to talk to businesspeople and get consensus on data definitions, business transformations or performance metrics. Don’t fall for it. Like the subprime fiasco the financial industry is facing, it includes false expectations based on faulty assumptions made by people who do not understand the consequences or won’t be around to suffer from them.

Year in and year out, studies keep showing that a majority of DW and business intelligence (BI) projects fail. These failures are not because of faulty technology but due to the inability to meet business expectations. To add insult to injury, most projects are late and over budget. The business believes the pitch that this next DW/BI project and tool will be the one to break through and create that “single version of the truth.” Buy the CPM, MDM, SaaS, SOA, DW appliance product, and your problems will be solved. How long will business users keep falling for those lines? How long before IT realizes that the data shadow systems that the business builds are the symptom of failed DW/BI efforts? DW and BI efforts may have terabytes of data and hundreds of business users on dashboards, but if businesspeople are still debating different numbers on their reports and creating these data shadow systems to plug the information gaps, then the efforts have failed to meet expectations.

How Do We Change?

Let’s start with the recognition that data integration has to do more with people, process and politics than with product. There are many terrific products available in the market today, but what we need first is to establish a framework to use them. Businesspeople have to help define and agree on data definitions, business rules (transformations) and performance metrics. That is the only way to enable enterprise-wide, consistent, comprehensive and quality data. This involves talking to businesspeople and achieving consensus across different business groups. That’s data integration the old-fashioned way. Once you have that, you can select from a palette of technologies and tools to integrate the data.

The work does not end when the latest version of the DW is in production, but involves an ongoing set of processes, such as data governance, to reach data nirvana. Your business and its performance metrics change as should your DW/BI solution. Failure to keep the DW/BI solution up to date guarantees it will be irrelevant at best and dangerous (providing wrong information to the business) at worst.

What’s Next?

Too often technology vendors will pitch their tools as the way to get you to nirvana without all the mess and fuss. Sorry, you cannot avoid that hard work. Establish and continually enhance your DIF if you wish to provide the business with the information it needs.