Portals eNewsletters Web Seminars dataWarehouse.com DM Review Magazine
DM Review | Covering Business Intelligence, Integration & Analytics
   Covering Business Intelligence, Integration & Analytics Advanced Search

View all Portals

Scheduled Events

White Paper Library
Research Papers

View Job Listings
Post a job


DM Review Home
Current Magazine Issue
Magazine Archives
Online Columnists
Ask the Experts
Industry News
Search DM Review

Buyer's Guide
Industry Events Calendar
Monthly Product Guides
Software Demo Lab
Vendor Listings

About Us
Press Releases
Advertising/Media Kit
Magazine Subscriptions
Editorial Calendar
Contact Us
Customer Service

Real-Time Data Warehousing:

    Column published in DMReview.com
March 11, 2004
  By Simon Terr

One of the things that I keep seeing over and over again is the implementation of projects using inappropriate technology. More often than not, these projects end up with, for lack of a better term, confused systems as they meet some needs but not others. The thing that keeps these systems alive is the substantial investments of time and money. Even then, after awhile the money stops playing a role and these systems end up ignored or replaced. To ensure that real-time data warehousing endeavors don't suffer the same fate, it's important to recognize where it makes sense to implement them and, equally important, where not to implement them. I've found that real-time data warehousing makes sense in two primary areas: operational and business intelligence applications.

Operational Applications

Contrary to popular belief, successful data warehouses are utilized for operational applications. Billing and usage reporting applications seem to be the first operational applications to take advantage of real-time data warehousing at places such as credit card institutions and telecommunication companies. To keep up with the billing and usage volumes, these companies have been forced to migrate from nightly batching jobs to real-time or near real-time data processing. Data warehouses are the natural choice for this type of processing as aggregation and fast retrieval of large amounts of data is what they do best.

Other operational applications include inventory management and service provisioning, to mention a few. Many businesses have moved to just-in- time inventory, so timing is everything. Once provisioned for some service, the billing clock starts ticking so tracking when the service was provisioned is significant. The temporal element is the common denominator for operational applications that can benefit from real-time data warehousing.

Business Intelligence Applications

One of the current trends in business intelligence is the implementation of corporate dashboards. The sophistication of corporate dashboards is quite broad and ranges from a report that lists several key performance indicators (KPIs) to dashboards that mimic the look and feel of automobile dashboards right down to the round gauges with needles that fluctuate in real time. Timeliness of the dashboards is driven by the data freshness in the data warehouse. Dashboards that display real-time data activity typically have a real-time data warehouse behind them.

The kind of information reported by these dashboards is as varied as the dashboards themselves. Financial information typically takes first dibs followed by trend and operational information. Financial information includes up-to-date revenue, current exchange and interest rates, and stock prices to mention a few KPIs. Trend reports typically display KPIs across some predefined timeframe, such as rolling 24 hour view, while operational KPIs can include equipment utilization rates.

Customer relationship management (CRM) business intelligence is an area that has been getting quite a bit of attention and where real-time capability is a value add, especially in a Web environment. To illustrate this, consider an established retailer who wants to see whether launching a Web-based discount campaign will increase sales while driving additional traffic to their Web site. A real-time data warehouse that captures and analyzes the clickstream and sales activity from the Web site can provide the information necessary to determine the effectiveness of the discount campaign.

The Internet is definitely an area that can and should reap the benefits of real-time data warehousing. Clickstream data generates substantial transaction volumes that, in popular sites, can easily eclipse credit card transaction volumes. These data volumes alone justify having real-time data warehousing. From a business intelligence perspective, for clickstream data to make sense, it has to be aggregated into predefined buckets to allow for click activity analysis. Once again, data warehousing is the natural choice for this type of processing.

When Not to Implement Real Time

The trick with real-time data warehousing is to know when not to implement it. For example, I recently developed a data warehouse that relies on transactional data from several external sources. One source provides transactional data on a daily basis whereas the other sources provide data on a monthly basis. Although the client is interested in having real-time data, it simply does not make sense to enable a real-time capability in this case. The data is only available once a day at best, so having the infrastructure to process the data in real time is overkill and adds an unnecessary level of complexity.

In addition to data availability, the business itself dictates whether real time makes sense. Take, for example, the automotive world where dealer inventory plays a role. If the automobile manufacturer expects a dealer to have 60 days of inventory on average, does it make sense to put in the data warehouse a subject area for automobile dealer inventory that gets updated in real time? The answer to that depends whether there is benefit in reacting to this information within hours of seeing it. Unfortunately, there are no fast rules about when not to implement real-time data warehousing. However, this does not mean that taking a step back and seeing if the endeavor makes sense in the larger scheme of things is not useful.

Implementing real-time data warehouses makes sense in operational applications where either large data volumes or time play a significant role. Business intelligence applications, such as corporate dashboards and CRM and Web tracking applications, also benefit from having real- time data warehousing. On the other hand, data availability and the business itself dictate when it does not make sense to implement real-time data warehousing. All in all, knowing when to implement real-time data warehousing is as important as knowing when not to do it.


For more information on related topics visit the following related portals...
DW Design, Methodology and Real-Time Enterprise.

Simon Terr is CEO and founder of KnowledgeSoft, Inc., a provider of products and services in the areas of real time data warehousing and business intelligence. Simon Terr can be reached at knowledgesoft@att.net.

Solutions Marketplace
Provided by IndustryBrains

KOM Networks Archiving and Data Storage
KOM Networks, a leader in archiving and data storage for more that 37 years, offers organizations a cost effective means to secure their growing data stores.

Speed Databases 2500% - World's Fastest Storage
Faster databases support more concurrent users and handle more simultaneous transactions. Register for FREE whitepaper, Increase Application Performance With Solid State Disk. Texas Memory Systems - makers of the World's Fastest Storage

Design Databases with ER/Studio: Free Trial
ER/Studio delivers next-generation data modeling. Multiple, distinct physical models based on a single logical model give you the tools you need to manage complex database environments and critical metadata in an intuitive user interface.

Validate Data at Entry. Free Trial of Web Tools
Protect against fraud, waste and excess marketing costs by cleaning your customer database of inaccurate, incomplete or undeliverable addresses. Add on phone check, name parsing and geo-coding as needed. FREE trial of Data Quality dev tools here.

Free EII Buyer's Guide
Understand EII - Trends. Tech. Apps. Calculate ROI. Download Now.

Click here to advertise in this space

E-mail This Column E-Mail This Column
Printer Friendly Version Printer-Friendly Version
Related Content Related Content
Request Reprints Request Reprints
Site Map Terms of Use Privacy Policy
SourceMedia (c) 2006 DM Review and SourceMedia, Inc. All rights reserved.
SourceMedia is an Investcorp company.
Use, duplication, or sale of this service, or data contained herein, is strictly prohibited.