Portals eNewsletters Web Seminars dataWarehouse.com DM Review Magazine
DM Review | Covering Business Intelligence, Integration & Analytics
   Covering Business Intelligence, Integration & Analytics Advanced Search

View all Portals

Scheduled Events

White Paper Library
Research Papers

View Job Listings
Post a job


DM Review Home
Current Magazine Issue
Magazine Archives
Online Columnists
Ask the Experts
Industry News
Search DM Review

Buyer's Guide
Industry Events Calendar
Monthly Product Guides
Software Demo Lab
Vendor Listings

About Us
Press Releases
Advertising/Media Kit
Magazine Subscriptions
Editorial Calendar
Contact Us
Customer Service

Data Integration:
The Advent of E3

online columnist Greg Mancuso and Al Moreno     Column published in DMReview.com
March 10, 2005
  By Greg Mancuso and Al Moreno

A few months ago we ran a series of columns dedicated to defining each of the major disciplines of data integration: extract, transformation and load (ETL); enterprise application interchange (EAI); and enterprise information integration (EII). We also asked for input as to which method or methods of integration are in use, or planned to be used, in your organizations. We received quite a few responses to this query; however, most of the responses raised more questions and they provided insight into current trends. The e-mails sent to our mailbox ranged from questions such as, "Why does the industry need differing methods of integration?" and "When should each method be used?" We also received several comments that point to a general misunderstanding or distrust in the "new kid on the block" - EII. Therefore, let's delve a little deeper into today's integration environment and talk a bit about E3.

META Group has recently coined the terms "E3" and the "E3 technologies." E3 is the idea that ETL, EAI and EII all play vital and complimentary roles within the integration landscapes we are sowing in today's corporate IT organizations. While it is true, it is very easy to state that no one of these technologies is the "right" way to build your integration practices. It is also true that the costs of deploying all three of these techniques into your organizations is a very expensive proposition and is, therefore, not right for all organizations. This month, we will take a very high level look at the three "Es" and define the roles at which they are best suited.

The granddaddy of the integration methods is ETL. The first real integration tools, those that came after the code generator phase, were ETL products. These tools were, and are, designed to work best in batch-oriented implementations. These tools are built around the premise that the target of the integration process is a database and that most ETL procedures will be run in a batch mode. While the target could be the operational database underlying a custom legacy application, the most common ETL target is a data warehouse, data mart or operational data store. ETL tools thrive at combining the dissimilar information from many legacy and operational sources and applying many complex data transformation and cleansing tasks with the intention of consolidating the information into a consistent and integration form that is of use by the organization's decision supporting applications.

Next off the boat was EAI. EAI is very much like ETL in that data can be sourced from many disparate sources and the EAI tasks may be run in batch mode. But, this is where the similarity ends. EAI tools are designed with the target of the processes being an application, not a database. EAI deployments are built around business process automation and the data is pushed from source application(s) to target application(s) via message traffic. The EAI tools use the applications' client interfaces or application programming interfaces (APIs) to gather and deliver the information. And, while they are often capable of performing some of the cleansing and standardization actions which are the bread and butter of the ETL tools, they are not nearly as efficient at these tasks.

And finally, we come to EII. The concept of EII is a few years old now, but the technology - both hardware and software - is finally catching up to the concepts. Now, we are just seeing the realization that information can be feasibly delivered to users without the need to perform the entire up-front integration activity. That is not to say that EII can or will supplant ETL or EAI in all IT shops. In contrast to both ETL and EAI, EII is designed from the ground up with the user in mind. The user is the target of the integrated information, and the user initiates the request for information in real time and in an ad hoc query fashion. EII tools are able to integrate information from various operational and decision supporting (data warehouse, data mart and/or operational data store) systems on demand and present a consolidated view of information, ranging from historical trends with related current data within seconds or minutes of when the user thinks up the needs for the information.

We said that there is room in the integration landscape for all three technologies, but where does each play? As can be inferred from the previous paragraphs, the ETL tools are quite well grounded in the high-volume and relatively static (predefined) data requirements. EII tools, on the other hand, thrive in the situations where the data requirements are not well defined or change on a frequent basis. They also work better in situations where the volume of returned data is relatively small. It's the middle ground between these extremes where the debate as to which technology is better begins. The easy - and best - answer is that neither technology is better. In many cases, these middle ground implementations are best answered with a hybrid ETL/EII solution. EAI will remain in the picture for the foreseeable future, but it is likely that the cases where it is the best choice will diminish over time.

Under what circumstances would each technology would be considered the "best bet?" Figure 1 points out some key pros and cons for each method.

Figure 1

As you can see, each technology has an important role to play in the integration frameworks you define for your organizations. And, each technology carries with it some baggage. What worked yesterday will likely work tomorrow. But, is there a better mousetrap? There is no one best way for all. And, you can satisfy your users' data needs without deploying all three technologies. The only hard and fast rule to data integration today is that there are alternatives.


For more information on related topics visit the following related portals...
Data Integration, Enterprise Application Integration (EAI), Enterprise Information Integration (EII) and ETL.

Greg Mancuso and Al Moreno are principals with Sinecon, a business intelligence consultancy specializing in data integration and BI/DW solution architecture design. Together they have more than 29 years of data warehouse and business intelligence experience and have implemented many large-scale solutions in both the U.S. and European markets. They may be reached at gmancuso@sinecon-llc.com or amoreno@sinecon-llc.com.

Solutions Marketplace
Provided by IndustryBrains

Design Databases with ER/Studio: Free Trial
ER/Studio delivers next-generation data modeling. Multiple, distinct physical models based on a single logical model give you the tools you need to manage complex database environments and critical metadata in an intuitive user interface.

Data Quality Tools, Affordable and Accurate
Protect against fraud, waste and excess marketing costs by cleaning your customer database of inaccurate, incomplete or undeliverable addresses. Add on phone check, name parsing and geo-coding as needed. FREE trial of Data Quality dev tools here.

Free EII Buyer's Guide
Understand EII - Trends. Tech. Apps. Calculate ROI. Download Now.

dotDefender protects sites against Web attacks
30-day evaluation period for dotDefender, a high-end cost-effective security solution for web servers that protects against a broad range of attacks, is now available. dotdefender supports Apache, IIS and iPlanet Web servers and all Linux OS's.

Click here to advertise in this space

E-mail This Column E-Mail This Column
Printer Friendly Version Printer-Friendly Version
Related Content Related Content
Request Reprints Request Reprints
Site Map Terms of Use Privacy Policy
SourceMedia (c) 2006 DM Review and SourceMedia, Inc. All rights reserved.
SourceMedia is an Investcorp company.
Use, duplication, or sale of this service, or data contained herein, is strictly prohibited.