DM Review Published in DM Review in August 2004.
Printed from

Business Intelligence: Recent Influences, Trends and Innovations

by Jonathan Wu

Summary: The future of business intelligence is very promising. This article discusses the business intelligence trends that will significantly benefit information consumers, as well as influencing factors and organizational issues surrounding them.

It's clear - the future of business intelligence (BI) looks promising, as evidenced by a very simple fact: more organizations are expanding their use of BI or are at least considering it. META Group expects the BI software market to grow 10 to 15 percent annually. To understand and appreciate current BI technology trends and place them into context, an understanding of influencing factors and organizational issues is helpful.

Influences on BI

Despite the significant economic improvement during the last 12 months, the prior three-year period impacted the way BI software vendors viewed their customers and enhanced their products. Before 2000, organizations purchased BI software in hopes of creating value or solving a problem. Then, the economy soured. Fortunately, BI was one of a few technology sectors that did not experience a dramatic reduction in spending. However, the sales cycle became much longer as potential buyers wanted to understand the associated benefits, return on investment (ROI) and total cost of ownership (TCO) of a BI solution. The longer and more challenging sales cycle forced BI software vendors to evaluate their product offerings and instill a greater discipline in their sales process.

The longer sales cycles and limited availability of capital contributed to the decline in the financial viability and demise of many BI software vendors. However, interestingly, there is still not one dominant product suite in the marketplace. This has left customers with numerous options.

In this situation, both large and small BI software vendors have a chance at winning a deal. The competitive environment has caused BI software vendors to reflect upon their market position and future outlook. The spreadsheet software market offers an example that BI vendors might be wise to learn from. In the 1980s, one could select from numerous spreadsheet programs. As the technology advanced and larger players such as IBM and Microsoft entered the scene, the number of spreadsheet programs declined rapidly to the point where Microsoft Excel emerged as the dominant spreadsheet program - and its leadership status remains today.

The market for BI applications, like the market for spreadsheet software in the 1980s, is still fragmented with numerous vendors - and the opportunity exists for a few to become the industry standard. The desire of BI software vendor management teams to develop into the dominant player in the field has led to a flurry of consolidations such as Business Object's acquisition of Crystal Decisions and Hyperion's acquisition of Brio. These two major transactions, both within the last 13 months, have helped these companies enhance their product suites and offerings. They are also a reflection of a maturing industry given the stabilization and usage of the technology, but one still having significant growth potential. After all, BI has not yet penetrated every organization.

The trend toward standardizing on a couple of BI tools is also contributing to market consolidation. In other words, BI vendors are trying to pack as much functionality into their product suites as possible in hopes of being chosen as the standard.

The direction of BI technology has shifted during the last several years. Prior to the economic downturn, numerous BI software vendors approached new product development as if it were a field of dreams; the "if you develop it, they will buy" attitude was evident in offerings such as mobile BI, desktop data mining for the masses and packaged analytic applications. This approach to product development relies upon market acceptance, which has shifted from vendors leading the direction to customers. Promises of value creation or problem solving have been replaced by business cases with clearly defined benefits and assumptions. Customers now drive the direction of BI technology more than ever before. Successful BI software vendors understand this shift and are focusing on providing comprehensive query, reporting and analytic capabilities with mass appeal that integrate well with existing infrastructure.

Legislative acts and international accords are also influencing the BI market and helping to create demand for BI technology. In recent years, several enactments have required managers and their organizations to provide greater monitoring of activities, reporting to external parties, privacy and safeguarding of data. Some legislative acts are industry specific, such as the Health Insurance Portability and Accountability Act of 1996 (HIPAA), which protects patient privacy and record security. HIPAA has four primary objectives: assure health insurance portability by eliminating job-lock due to preexisting medical conditions; reduce healthcare fraud and abuse; enforce standards for health information; and guarantee security and privacy of health information. This United States legislative act directly impacts the healthcare and pharmaceutical industries.

The banking and finance industry, meanwhile, is affected by the Basel II Capital Accord, which was established by international banking supervisors. The goal of Basel II is to improve the consistency of capital regulations internationally, make regulatory capital more risk sensitive, and promote enhanced risk-management practices among large, internationally active banking organizations.

Some legislation, such as the Sarbanes-Oxley Act of 2002, impacts all publicly traded companies, regardless of industry. This act was intended to bolster public confidence in the United States capital markets by imposing new duties and significant penalties for noncompliance on public companies and their executives, boards of directors, independent auditors, legal counsel and securities analysts. These legislative acts and international accords are just a few examples of the numerous enactments that managers of organizations must address every day. While each of these enactments serves a specific purpose, they all have one thing in common: the need to manage and report upon information.

These influences have created a greater awareness and appreciation of information that, in turn, is boosting broad demand for BI technology. However, each organization has its own unique information challenges.

Organizational Learning

The recent economic downturn exposed questionable business models and practices, policies, procedures and expenditures. In addition, the downturn brought to light the greater need for information. Every manager was forced to examine and assess his or her own organization. An essential component to making informed decisions is having relevant, timely and accurate data at your disposal. Those management teams that effectively addressed their organizational deficiencies were able to make informed changes, pull through the economic downturn and position their organizations for success as the economy turned around. This has enticed organizations to improve their information systems for monitoring, reporting, analysis and decision-making purposes.

However, technology by itself is not the answer to solving business problems or adhering to government regulations. Gone are the days of purchasing software without clear requirements and objectives. Also gone is the purchasing of excess licenses or software products that become shelfware. Excess expenditures were eliminated during the economic downturn.

Organizations have learned that one must understand the business problems, processes and current information infrastructure in order to address the business problems. The organization's ability to absorb new technology must also be considered. Significant challenges will be encountered when attempting to move from manually produced spreadsheet reports to an information portal without effective change management, which provides incentives that induce individuals to embrace new practices, procedures and technology. However, there is an inherent limitation on individuals' ability to quickly develop or comprehend BI information solutions and enact change unless they have the necessary technical background and business acumen. If individuals do not have the experience or skills, the learning curve is extremely steep and more time is needed to develop the appropriate knowledge.

However, the progression from spreadsheet reporting to a robust BI environment is not just based upon change management or individual aptitude. James Cates, author of the Ladder of Business Intelligence (LOBI) framework, defined levels and activities for individuals to address if they want their organization to mature in its use of BI. The LOBI framework contains six levels in which an organization can progress from Level 1, relying on facts that are created by spreadsheets or disparate data sources, to Level 6, enabled intuition that is created by deep insight. The progression from one level to the next requires repeated and systematic execution of business role identification, business process identification and technology selection. Understanding where your organization is in the LOBI framework positions you to then consider the activities, processes and technology needed to progress to the next level.

Those organizations that have successfully implemented BI realize the investment they have made and the value it provides. A number of our clients have created or are considering BI centers of excellences (CoEs) as a means of maintaining and evolving their BI environment. BI CoEs have also facilitated changes that elevate organizations to the next level in the LOBI framework.

There is a greater recognition and understanding between satisfying business needs and the capabilities of BI solutions. In order to create value for the organization, expectations must be managed and information consumers must be using the BI solution.

BI Technology Trends

Both external market influences and reflective organizations are directing the BI trends observed today - and those with sufficient market demand and appreciation are discussed in detail in the following sections.

Performance Management

Wayne Eckerson of The Data Warehousing Institute defines business performance management (BPM) as, "A series of processes and applications designed to optimize the execution of business strategy." Gartner believes that through 2005, 40 percent of enterprises will adopt performance management. BPM's role is to provide individuals with information they need to manage the activities for which they are responsible. Through the combination of business process, transactional systems, data warehousing and BI, BPM can be created to enable a closed-loop decision-making process. In BPM, the process begins with defining the strategic goals and objectives of senior management. From there, plans are created that are then used for the tactical execution of activities. Those activities are monitored, the results are assessed and then adjustments are made to refine strategy, plans and execution to ensure the organization is moving in the desired direction.

Gartner's Second Quarter 2004 Magic Quadrant for corporate performance management software vendors shows that no product has yet entered the leader quadrant, although several software vendors are close to doing so. The economic downturn has reminded business leaders that success is based upon informed business decisions as well as the ability to monitor the results of those decisions. Legislative acts and accords are also requiring managers to more strongly control their business processes and practices while safeguarding data and reporting to external constituencies. These two influences are creating the demand for performance management solutions.

Integrated Platforms Between Product Modules

Through mergers and acquisitions, large organizations have collected a variety of BI products. Some organizations have attempted to standardize on one particular product suite. This is always a challenge, though, because people tend to bond with their favorite BI products. Consider a large healthcare products distributor that uses BI software products from seven different vendors. Each serves a specific purpose and is championed by a business sponsor. While standardization is a goal of theirs, it will be difficult to achieve without creating great discontent. As a result, these organizations are aiming for the best product in a category because no single vendor offers the best suite of products that contains ad hoc and structured queries, standard reporting, OLAP, exception-based reporting, data mining and analytics capabilities. Gartner says that "most enterprises will need more than one BI tool and must select them as part of an integrated decision within the context of a BI framework." As such, issues associated with data definitions, data conformity and comparability will continue to create inefficiencies as information consumers work with multiple BI products. Unfortunately, this issue will not be resolved until there is a dominant BI vendor.

Structured and Unstructured Data

Most BI products only process structured data captured by another application or transactional system, such as customer orders entered and processed by an enterprise resource planning (ERP) application. However, a few BI products are capable of also integrating unstructured data such as documents and images. Organizations capture tremendous amounts of unstructured data, such as e-mail and mail correspondence, every day. META Group estimates that, on average, 70 percent of an enterprise's information assets are in an unstructured form. The value of unstructured data has been limited due to its form and the challenges of integrating it with structured data.

For example, conducting an analysis of your most profitable customers will yield a list of customers and their characteristics. If this information is used to establish a threshold for your customer base, you risk eliminating unprofitable customers who could eventually become profitable customers. Information regarding the potential of unprofitable customers may be contained in an unstructured format such as correspondence or news briefings. The opportunity exists to provide individuals with greater insight into subject areas when both structured and unstructured data are presented.

Data Quality

This is not a new topic; however, poor data quality remains the bane of BI solutions. Gartner believes that "through 2007, more than 25 percent of critical data within Fortune 1000 enterprises will remain flawed - that is, inaccurate or incomplete." Poor data quality leads to discontent among information consumers and eventually to limited or discontinued BI usage. While customers struggle to address this problem, BI software vendors are well positioned to assist through data profiling, validation and notification capabilities. By incorporating these capabilities, BI solutions can highlight data problems or validate the accuracy of the data while the information consumers are reviewing it. Corrective steps can then address poor data quality while building tremendous confidence in the BI solution. In addition, these capabilities would alleviate discontent among information consumers as poor data quality is actively brought to their attention. This benefits information consumers and BI vendors alike.

Predictive Analytics

Sophisticated BI users need predictive analytics to model "what if" scenarios. For example, someone might analyze sales by geographic region and time period for the prior quarter. That information can then be used to forecast sales for future periods given the historical trends. While query and reporting capabilities provide data on historical events and transactions, the ability to perform predictive modeling, analysis or analytics is limited: much more functionality is currently provided by spreadsheet applications. Because BI applications have the ability to access and present information, predictive analytics is a natural extension. However, META Group says that predictive analytics are increasingly being embedded into enterprise applications (customer relationship management, supply chain management, etc.) instead of being purchased as standalone data mining solutions.

Refined Functionality and Capabilities

Spreadsheet programs are widely used by organizations of all sizes because they are intuitive and downright easy to use. Gartner believes the majority of BI users in an enterprise are "occasional information consumers" who only need basic reports and spreadsheet functionality. Unfortunately, the features and functionality of most BI programs are far beyond the abilities or understanding of those who typically work with them. As such, the ability to correlate classes of information consumers to defined levels of features and functionality is needed. This approach provides individuals with the appropriate capabilities, helping to develop their skills. A result of this approach is also the natural progression and development of highly valued knowledge-workers.

The convergence of economic conditions, legislative enactments and advancements in BI technology presents a wonderful opportunity for BI software vendors to prosper. Learning from past lessons and historical events in technology, leading BI software vendors are racing to become the industry standard through expansion of user licenses and product suites. These events are influencing the technology trends in BI, which significantly benefit information consumers. The ability of organizations to realize the value from their investment in BI has never been better.

Suggested additional reading:

1. Lane, Dean. CIO Wisdom: Best Practices from Silicon Valley. Prentice Hall PTR, 1st edition (July 25, 2003). Chapter 16: Ladder of Business Intelligence: A Systematic Approach to Success for Information Technology, pp. 373 - 394.

Jonathan Wu is a senior principal with Knightsbridge Solutions,, the largest independent professional services firm specializing in business intelligence and data warehousing. He has extensive experience designing, developing and implementing information solutions for reporting, analysis and decision-making purposes. Wu initiated and contributed to the development of the Business Intelligence and Data Warehousing Certificate Program at the University of California, Berkeley Extension and is also a lecturer in the program. In addition to this quarterly column, Wu is the business intelligence online columnist for He can be reached at

Copyright 2005, SourceMedia and DM Review.