Right Time for Real Time?
“It was so real. It was really... it was real; it was realistic.”
- Otto, in Michael Nesmith’s “Repo Man”
Is real-time business intelligence really all that real? Is it indeed the right time for real time? A recent report by The Data Warehousing Institute (TDWI), entitled “Building the Real-Time Enterprise”, provides ample evidence that yes, real-time BI really is for real. Among the report’s findings:
- Of the 846 respondents to the report’s survey, half said they had either implemented a real-time project or planned to do so;
- Of the 108 people who said they had deployed a real-time project, four out of five said it was successful, with 42 percent reporting fair success and 38 percent calling their project very successful;
- Of the 313 individuals who said they planned to implement a real-time project, more than half (56 percent) said they expected to do so within one year.
Such success stories beg the question: what exactly constitutes a real-time enterprise? Colin White, president of BI Research and author of TDWI’s report, explains: “The term ‘real time’ has become synonymous with an organization’s ability to become more agile and thus more responsive to changing business circumstances. Although this agility undoubtedly can provide business benefits, there is still universal confusion about what real-time BI means from both a business and a technology perspective.”
Survey participants were given a list of possible definitions for real-time processing – from both technical and business perspectives – and were then asked to pick the three which they considered most accurate. The results demonstrated a wide range of opinions as shown in Figure 1. The top three answers were: real-time generation of reports and analytics (53 percent), faster decision making (52 percent) and business activity monitoring (48 percent). Close behind were: business user alerting (45 percent), operational real-time reporting (42 percent) and real-time maintenance of a data warehouse (36 percent). Further down the line were: automated recommendations and actions (24 percent), split-second query response times (13 percent) and other (3 percent).
Figure 1: Some Definitions for Real-Time Processing
Worthy of note is that definitions relating to system performance scored lowest, seeming to indicate a near-universal understanding that real-time BI is much more about business needs and results than it is about hardware or software.
Right Time for Right-Time
Such is the philosophy of real-time data warehousing pioneer Stephen Brobst, a contributor to the report and frequent instructor at TDWI's quarterly conferences. "The first thing to realize is that right time is more important than real time. The goal should be to deliver the right data to the right person at the right time," says Brobst, managing partner for Strategic Technologies & Systems. "Real time means something very specific to me – and most data warehouses are not even up to date at the level of near real time. The key is that the business drivers must dictate the data freshness. For some decisions it might be hourly, some up to the minute. So it's clear that if you define not in terms of real time but right time, you'll be on the right track.”
The report mirrors this thinking. Says White, “When developing real-time applications, organizations need to consider how responsive each business process must be to satisfy a specific business goal. From a business perspective, it’s more useful to think in terms of ‘right time’ rather than ‘real time.’
Depending on business needs, the action-time requirements in real-time environments may vary from a few seconds to several hours. A practical definition, therefore, for the meaning of real time is ‘the ability for an organization to react to business needs and changing business circumstances within a single day.’”
Four Types of Real-Time Processing
Still, additional layers of complexity cloud the dialogue about what real-time means from one company or industry to another. The report outlines four types of real-time processing occurring in the market today:
- Straight-through processing (STP) is when a business transaction is entered once and processed to completion, thus eliminating the need to reenter data or to batch transactions for unnecessary processing. STP benefits an enterprise by reducing delays and improving the flow of key business processes. An example of effective STP exists in some online retail operations where transactions are immediately processed such that inventories are maintained in real-time; this prevents customers from ordering products that are no longer in stock, which sometimes happens when online retailers use a traditional batch process for supply chain management. STP usually makes extensive use of application integration software.
- Real-time data on demand provides both business users and applications with access to consistent, current data with which to make intra-day decisions. The report notes that many companies are capturing data from multiple operational systems and integrating it into a low-latency store for operational reporting and analysis. Some survey respondents also reported using a low-latency store to consolidate operational reference data, in some cases to propagate data between systems (such as front office to back office).
- Real-time performance management enables a BI system to monitor and measure business performance by tapping directly into business events as they flow through the enterprise. Measurements are gauged against specific goals and objectives with alerts sent to appropriate personnel and/or systems when predefined limits are reached – such as inventory levels or other situations that require attention. Gartner Research has coined the term business activity monitoring (BAM) to describe this type of processing. Several vendors now offer BAM solutions. Gartner predicts that BAM will be one of the top four issues driving IT in 2004 for enterprises in which faster reaction is key to operational effectiveness.
- Real-time predictive analysis, unlike the first three types of real-time processing listed, is proactive in nature, not reactive. Predictive analysis tries to determine the outcome of business events, such as a loan going bad or a credit card transaction being fraudulent. This type of processing involves information being sent in real-time to a predictive analysis application, which plugs the data into a “black box” that applies various rules or models (algorithms) to determine a recommended action. Such rules or models can be generated from historical business intelligence via the use of data mining tools, for example.
Proof is in the Process
Regardless of how well a real-time BI system is designed or maintained, bottom-line value hinges on an enterprise’s ability to adapt. Essential for that ability is a clear understanding of business processes. The report notes that business process management is now consequently receiving heightened attention. Says White, “Process management software helps an organization analyze, document and manage its business processes, whether they be simple processes such as handling customer orders or more complex processes such as matching supply and demand.” The report also notes that many application integration software vendors are either developing process management solutions or buying them.
So important is this understanding of processes that White punctuates the report with a process-management-centric conclusion, quoting Jim Cates, CIO and vice president of IT for Brocade Communications Systems: “To be successful in real-time projects you need to understand your business processes, the roles associated with those processes and the information that is required for each role. More data doesn’t lead to better decisions. I often think the reverse is true ... From a technology perspective, you need to understand the impact of data on the business process before collecting it and need to ensure that the data sources used to create information are accurate enough to make decisions in real time.”
Balancing Benefits and Costs
Of course, every company must work within a budget in order to remain solvent. While real-time BI can enable an organization to respond more quickly and accurately to changing circumstances or new opportunities, the cost structure for implementing such a system can be prohibitive. Any resources invested must be balanced against resulting ROI. In offering insight on how to determine where the sweet spot is (the overlap between affordability and the value generated by real-time functionality), the TDWI report references a concept coined by Richard Hackathorn, president of Bolder Technology:
“There will always be delay between a business event occurring and appropriate business action being taken,” he says. “In a BI environment, this delay has three components: data latency, analysis latency and decision latency.” The sum of those latency periods is the action time, and the objective of any real-time system is to reduce the required action time. Whereas IT solutions can reduce data and analysis latency, any reduction in decision latency depends on the business user.
The report recommends several options for reducing this decision latency. Says White, “Providing the user with better information will of course help reduce decision latency. Rather than simply telling users a problem has occurred, for example, it is of more value to also provide detailed analyses and suggest actions. Another approach is to replace manual user decision-making with rules-driven BI processes that automatically take action on behalf of the user.”
The report provides solid testimony that real-time BI can be extremely valuable. “The benefits are unbelievable if you do it right,” says Danny Siegel, senior manager of Business Technology at Pfizer Inc. “With real-time BI you can drill down from strategic data to what happened 15 minutes ago.”
Brocade’s Cates takes a pragmatic approach to this issue. “I intend to invest in solutions that enable me to provide answers to the top 10 business questions for each business role of a business process. If I can do this, then I know the investment will pay for itself.”
Cutting Edge
For a clear indication of why real-time BI is the very cutting edge of modern decision support, consider where it’s used. Brobst, who also serves as CTO for Teradata, explains. “Real time tends to occur in industries that are the most competitive. Retail in the U.S. is extremely competitive, so the top-tier players there have been dong it for some time. If I buy something at point of sale, for example, by the time the groceries have been loaded in the car, the data is in the data warehouse. In the Australian marketplace, retail is not very competitive but banking is more aggressive, so that industry is very active in real-time initiatives. And wherever telecom has been deregulated you see lots of activity as well. It’s really driven by competition.”
Of course, novelty carries risks as well as rewards. Real-time business intelligence is still fairly new, and as such presents the expected array of unknown hurdles. The report offers some practical advice on how to prepare for a foray into this realm of real time:
- First, it’s important to gain an understanding of real-time BI technologies and their business benefits. And even though latency requirements may range from seconds to hours, a clear business case and expected ROI must be established for projects. Consequently, IT managers must communicate thoroughly with business users to determine where real-time BI would provide the most benefit.
- Second, real-time applications must be integrated with the existing BI infrastructure. Without this integration, the end result with be another information silo. The complexity of accomplishing such integration should not be underestimated.
- Third, real time is happening in real time: it’s new and developing, which means there are risks not typically associated with more proven disciplines. Real-time BI programs should be catalogued carefully to enhance overall adaptability.
- Finally, technology is only part of the equation. End users must be courted and continuously educated as to the functionality at their disposal, while their needs must be heard. And the business itself must be flexible enough to capitalize on the intelligence generated by such solutions.
TDWI’s report also includes extensive information about how to build a real-time BI solution, how to prepare for real-time processing, and how to apply real-time BI to business problems. To learn more, download the full report at http://www.tdwi.org/display.asp?id=6841.
Industry Buzz
Discussion about real-time BI and the importance of BPM echoes throughout the greater business intelligence industry. Eric King is president of The Modeling Agency, a Houston-based firm that offers training and consulting in data mining. He says they’ve learned over the years to caution their prospects and clients about the critical importance of understanding business processes. “Companies that do not have a solid grasp of their processes are often ill-positioned to make use of the intelligence we provide them, and that will ultimately be a real pain point,” he says.
Steven Arbogast spent more than 30 years doing business process management consulting at IBM, and now heads up the U.S. operations of a Danish BPM software and consulting firm called QualiWare. He cites the rise of business intelligence as the driving force behind the resurgence of BPM.
“Managing business processes is an ancient practice,” says Arbogast. “What makes today’s BPM renaissance especially valuable – and this is why it’s getting so hot – is that BI has multiplied its value exponentially. Companies can now get quality real-time intelligence from a BI system, then act on that information the same day, effectively altering multiple business processes – all with a few clicks of a mouse. What once took days or weeks can now be done in seconds. And what we’re seeing today is just the beginning: as companies more thoroughly and seamlessly integrate their enterprise architectures, while at the same time cataloguing and analyzing their business processes, we’ll begin to see such intra-day activity becoming the norm. The end result should be a major surge in productivity.”
Guidance from Above: Sarbanes-Oxley
Another major factor pushing the deployment of real-time projects is the already infamous Sarbanes-Oxley Act of 2002. Norman Comstock is a practice director for Jefferson Wells, a national management consulting firm, who says much of his time is spent focusing on three sections of the act:
- Section 302 calls for a quarterly certification by the CEO and CFO regarding the financial condition and results of operations, fairness of accounting presentation, review and disclosure of significant deficiencies and material weaknesses in the internal controls. This section has been effective since the fourth quarter of 2002 and is an ongoing requirement.
- Section 404 calls for public accounting attestation and reporting on the internal control assessment made by management. The annual report shall contain an internal control report, which 1) states the responsibility of management for establishing and maintaining an adequate internal control structure and procedures for financial reporting; and 2) contains an assessment, as of the end of the fiscal year, of the effectiveness of the internal control structure and procedures of the company for financial reporting. This section will be effective in 2004 for many large public companies. Other public companies or companies with public debt are also subject but not affected until 2005.
- Section 409 addresses enhanced and real-time disclosure by issuers of securities. The deadline for Section 409 is not yet official.
Comstock notes that the nexus of these three sections of the Act creates the perfect storm for triggering a flood of real-time initiatives and innovation. While many public companies have ample documentation of controls, the infrastructure to inventory, monitor, audit and authenticate disclosure controls is not mature. Many small- and medium-size companies have substantial manual processes to facilitate compliance with section 302 and 404. Real-time disclosure could consequently be overwhelming without the help of integrated technology. Comstock says there are still critical questions that remain to be answered, however, such as the deadline, and what the working definition of “real time” shall be. “My customers are asking me about section 409 but are scrambling to complete the documentation and testing necessary to prepare for attestation under section 404,” he says.
Still, Comstock says this real-time regulation should help usher in a new era of enterprise integration, one in which BI and operational monitoring systems will become thoroughly intertwined. The point, he says, is that real-time needs will be facilitated by systematic tools in the background (namely, BI systems) that have been the traditional monitoring tools for key performance indicators. “BI systems therefore become the most logical vehicle to help address real-time disclosure,” he says.
Eric Kavanagh is web content developer for DM Review. Prior to joining DM Review, Kavanagh was web editor for TDWI. He also founded and ran Mobius Media, a strategic communications firm based in New Orleans.
