Real-Time Analysis and Data Integration for BI

Summary: While architecting the real-time decision support environment, business and technology managers must avoid the trap of integrating everything at once and making everything real time. Prioritizing the integration of crucial processes and information for making critical decisions more quickly is the key to successful real-time decisions.

Common sense is becoming uncommon in the business intelligence (BI) marketplace these days. In the rush to deploy BI solutions, business leaders and information technology (IT) managers are not applying due "intelligence" in determining which business intelligence solutions may benefit their business and how. As new BI concepts are surfacing, organizations are rushing to deploy them without performing up-front analysis. This lack of direction and game plan is leading organizations to unsuccessful BI projects, causing frustration and business ineffectiveness.

BI Evolution

BI is continuously evolving. We have seen BI evolve from ad hoc requests for sales and productivity reports in the 1960s to what we see in today's agile corporations that have combined different decision support techniques to continuously improve business performance and gain success (see Figure 1).

Figure 1: Business Intelligence Evolution
Pictures taken from http://www.wilderdom.com/evolution/ HumanEvolutionSequencePictures.htm

However, technology and business managers alike are struggling to identify and align the right decision support system for business from a plethora of available BI solutions. Before deciding on the BI solution, organizations must evaluate their needs for it.

Value of Different Types of Decision Support Systems

In any business organization, generally three types of decision making take place (see Figure 2):

1. Operational decision making to ensure that day-to-day operational systems are running efficiently.
2. Tactical decision making to ensure that existing operations and processes are in line with the organization's business objectives and strategies.
3. Strategic decision making to create a road map for continuous success and business growth.

Figure 2: Types of Analysis

The type of decision support system to be selected depends on the type of decision that is most critical to the organization. In an agile organization, operational decisions require access to the operational information in real time. Knowledge of available inventory, status of orders and number of support calls in the call queue of the support desk are required to prioritize production-line activities, order shipments and assign employees to the support desk calls.

On the other side of the spectrum, strategic decisions in any organization are normally based on historical, fairly static information because an organization, regardless of its agility, needs a well-planned strategy that does not change continuously. In the middle of the decision-making spectrum lies tactical decision making, which requires a combination of strategic and real-time information to forecast monthly and quarterly activities.

It is generally understood how conventional data warehouses and reporting tools help determine long-term and short-term strategic vision. However, the majority of organizations don't seem to have an appreciation for the importance or impact of real-time business analysis.

According to DM Review's 2003 Readership Survey, the primary objectives targeted with BI solutions are (respondents were asked to select all that were applicable):

• Increase customer satisfaction/retention ? 62%
• Decrease costs ? 53%
• Increase revenue ?­ 48%
• Increase profit ?­ 41%
• Increase market share ?­ 37%
• Provide product direction ?­ 30%

Real-Time Operational Decision Making

Unfortunately, scenarios that approximate the following occur almost every day, forcing organizations into losing situations that could easily be avoided. One hour after the accounting system becomes aware of a customer's bankruptcy filing, a large order of a product in high demand is shipped to that customer. The shipping system only becomes aware of the stop-shipment order the next day, after the nightly process has synchronized the accounting and shipping systems. Even though one department received business-critical information in real time, the order was still shipped to a possibly defunct customer. The likelihood that payment for the shipment will ever be received is low.

As this example illustrates, for any real-time operational environment, it is important to have in place the processes and technology required to act in real time. Fortunately, this can be achieved by simply automating the routine decision-making tasks that can solve common business problems by applying generally accepted procedures.

To automate these decision tasks, one must diligently look at frequently occurring predictive processes. Some of the examples of these time-dependent predictive processes are:

• Changing the price of a product
• Allowing a customer to overdraw
• Offering discount to a particular customer
• Offering specials to specific customers

In addition to providing the ability to automate these routine predictive processes, real-time analysis also extends the ability to automate decision tasks for routine problems that require non-routine treatment. For example, if a customer is rated as high risk, then supervisor-level authorization is required before an order can be shipped to that customer.

The conventional, historical-based data warehouses and decision support systems by themselves do not offer the possibility to automate "operational decision tasks in real time." This type of automation requires dynamic accessibility and the ability to process operational information.

Real-Time Enterprise

While architecting a real-time operational enterprise, business and IT architects must be very selective about the functionality of a real-time decision support solution. Too much real-time data will create unnecessary chaos that will blur the view of decision-makers at all levels.

As Figure 3 shows, the raw information from operational systems is cleansed and appropriately organized first into data stores such as operational data stores (ODSs) and data warehouses (DWs), and then into information stores such as cubes, reports and queries in order to prevent overwhelming the employees with information. This allows organizations to ensure that line managers are made aware of and react in real time to appropriate events, not just ad hoc business events.

Figure 3: Effective Decision-Making Environment

To change an organization's focus from "what happened" to "what is happening," organizations must start to transfer and process business information through automation and in real time. While architecting the real- time decision support environment, business and technology managers must avoid the trap of integrating everything at once and making everything "real time." Prioritizing the integration of crucial processes and information for making critical decisions more quickly is the key to successful real-time decisions.

Shaku Atre is the president of Atre Group, Inc., a consulting and training firm in Santa Cruz, Calif., that helps clients with business intelligence, data warehousing and DBMS projects. Formerly, she was a partner at PricewaterhouseCoopers and held a variety of technical and management positions at IBM. Atre is the author of six books, including Data Base: Structured Techniques for Design, Performance and Management and Distributed Databases, Cooperative Processing & Networking. She is most recently coauthor with Larissa Moss of Business Intelligence Roadmap: The Complete Project Lifecycle for Decision-Support Applications (Addison Wesley, 2003). Atre can be reached at shaku@atre.com. Dipendra Malhotra is the director of research and development at DataMirror. He is responsible for developing DataMirror's line of real-time data integration, ETL, wireless data collection and business performance monitoring software products and plays a key role in guiding the company's product strategy and market direction. Prior to joining DataMirror, Malhotra worked at Accenture and Bell Canada where he helped design and develop multiple OLAP, OLTP, wireless and data collection applications and products. Malhotra has a Bachelor of Engineering (Honors) degree in computer engineering from McMaster University, Canada and has presented at numerous industry conferences.

Copyright 2005, Thomson Media and DM Review.