FREE DM Review Site Registration!
Sign-up today and access DM Review on the Web!

Your FREE registration entitles you to:
FREE email newsletters

FREE access to all DM Review content

FREE access to web seminars, resource portals, our white paper library and more!

   

Technologies to Take Your Enterprise-Wide BI and DW Program to the Next Level

BI Solutions

My online series of columns is focused on the need for businesses to get serious about approaches to developing enterprise business intelligence (BI) and data warehouse (DW) capabilities. When pursuing these capabilities, it is important to adopt a holistic view followed by disciplined investment and execution. To develop future vision for this capability, consider seven interrelated areas:

 

 

This column explores additional consideration of the architecture focus area. It specifically addresses the role of key technologies that can take your enterprise BI/DW architecture to the next level.

 

Beyond the Basics

 

In my previous columns on enterprise BI/DW architecture, I reviewed the basic technologies individuals should cover and the impact of packaged applications on architecture. As BI/DW programs mature there are other technologies people should consider for their unique benefits such as:

 

  • Enterprise information integration (EII) tools,
  • Enterprise application integration (EAI) tools and
  • DW appliances.

Enterprise Information Integration

 

EII tools provide the ability to join data from multiple databases simultaneously. They can join data from multiple database management system (DBMS) vendors running on multiple physical servers. When some people first hear about EII tools, there is a temptation to think, “This is great. I don’t need to create a physical DW anymore; I can just create a virtual DW by joining all of my transaction systems.” This is a much too optimistic assessment, because integrating data from many disparate systems is difficult for many reasons. For example:

 

  • There aren’t always matching values to join on, which is why extract, transform and load (ETL) processes often incorporate quite a bit of data transformation and mapping.
  • Queries across multiple systems can have poor response times.

While the initial EII thought is tempting, it also ignores some of the strengths of a physically stored DW. For example:

 

  • A DW may store much more history than the transaction systems themselves.
  • A DW can be designed to store versions of the source data as it changes over time - not just what currently exists in the sources at a certain moment.
  • A DW can be optimized for speedy queries.
  • A DW can prevent source systems from getting bogged down while handling long-running queries of large volumes of data.
  • Along with marts and cubes, a DW can implement much more complicated calculations and business logic than a distributed query.
  • Along with marts and cubes, a DW can implement a data security scheme consistently across all subject areas regardless of the specific data security rules (or lack thereof) implemented in each particular source.

Despite these challenges, however, EII tools can be used to complement BI/DW solutions by allowing:

 

  • Historical data in a DW or mart to join with real-time data in a transaction system to support a reporting requirement and see information from different points in time all once,
  • The creation of a virtual data mart as a prototype to learn what the challenges of creating a physical DW or mart and will be.
  • Multiple data marts to be joined/federated without creating a new data mart resulting in much quicker delivery of the new solution if the data is already in multiple data marts.

Enterprise Application Integration

 

EAI tools are being used in many companies these days as a means to transport information from one business application to another and to better automate processes requiring business activities to be tracked across multiple systems.

 

As with EII tools, it may be tempting to think you no longer need ETL tools because you can just load your DW using EAI messages. Unfortunately, this presents several challenges:

 

  • EAI message buses may become overwhelmed and slow down with large data volumes that are often in a DW (which is contrary to the original intent of EAI systems - to quickly communicate critical information from one transaction application to another).
  • If you don’t need the information in your warehouse in a near real time, EAI may be an unnecessarily expensive option.
  • EAI experts are often busy working on application-integration tasks; so they may not have a lot of free time to work with the BI project teams.

Even though you will not want to eliminate ETL with EAI, you may want to use EAI in cases where you must load data in near real time to the warehouse to meet a particular reporting need. This strategy is often utilized when there is a reporting need to monitor a business activity.

 

Data Warehouse Appliances

 

This is another specialized technology category that may be desirable to consider when it is likely that the general purpose DBMS technologies are not going to meet the needs of a particular BI application or for the enterprise itself. Data warehouse appliances typically consist of specialized DBMS software in combination with specialized hardware. They are designed to be optimized for DW or data mart functions such as high data scalability, high user scalability, fast data loading and fast querying. This is more than just a bundling of hardware and software for marketing purposes. Rather, DW appliances are designed from the ground up to meet extreme DW needs. They can be more expensive than off-the-rack DBMS solutions, but sometimes are the only option for the levels of scalability and performance required for certain BI applications. Although the initial price tag can be more expensive, there is an argument to be made that DW appliances can reduce the overall cost of ownership with reduced costs for development, support and maintenance as compared to alternatives.


Robert Farris, Hitachi Consulting Vice President and Business Intelligence Capability Practice Leader, has more than 19 years of information technology and consulting experience. He has served both in consulting organizations with Andersen Consulting, Navigator Systems and Hitachi Consulting, and in industry organizations with Bankers Trust and American Power Conversion. Farris specializes in developing the strategy for a BI Program, specifically defining and implementing the team organization, architectures, technologies, methodologies and internal processes. Farris is a graduate from Purdue University with a Bachelor of Science in Industrial Management with a minor in Computer Science. He may be contacted at rfarris@hitachiconsulting.com.

For more information on related topics, visit the following channels:



Industry Vendors