FREE DM Review Site Registration!
Sign-up today and access DM Review on the Web!

Your FREE registration entitles you to:
FREE email newsletters

FREE access to all DM Review content

FREE access to web seminars, resource portals, our white paper library and more!

   

Managing BI Complexity

In A. J. Deutsch’s short story “A Subway Named Mobius,” Boston's Public Transportation authority introduces a new line, and the topology of the network becomes so complex that a train vanishes, lost in some fourth-dimensional properties of the network.

 

The mathematics in this story may not always be accurate, but many managers today can identify with the situation. They work with systems that are increasingly complex - where it may seem almost impossible to understand all the interconnections.  Many fear they may lose track of some important detail that could cause their train to jump the track.

 

Enter Business Intelligence

 

Business intelligence (BI) is supposed to address data and information complexity in the modern business environment to help minimize the decision-making complexity.

 

BI should:

  • Supply accurate data in a timely manner in a secure environment,
  • Present data in the form of easily digestible information,
  • Provide easy means to answer ensuing questions, and
  • Allow effective communication.

BI should simplify gathering and presenting information so that less technical knowledge is needed and communication between the members of the organization is improved. Information should be surfaced in common business language rather than cryptic database languages, thus keeping information retrieval and consumption simple for all.

 

Simple framework of BI means lesser introduction of “newer parts,” easier administration and maintenance. This also means more time and energy can be spent on analyzing the information than trying to gather, store and reconcile.

 

Real-World Challenges

 

Complexity challenges BI in various ways. Some companies, especially the large ones, experience multiple complexity challenges.

 

As organizations grow, they may end up with one or more transaction systems for a variety of reasons: mergers and acquisitions, additional applications due to growing needs, different software vendors, legacy databases, etc.

 

The first challenge is to access and combine data from these multiple databases in an efficient and timely manner. Redundant databases like the data warehouse or data marts impose additional “feeding and seeding” requirements that can be a big drain on resources.

 

These databases may be centralized or dispersed. Consolidation often poses a performance challenge. It is imperative to execute queries with minimal impact on system performance and with minimal hardware investment. The consolidation has to make use of every resource available – hardware and network.

 

Businesses often need on-demand business analytics – on-hand inventory from multiple locations, open orders or upcoming shipments to ship from various locations, etc. As businesses need to react more quickly to market demands, putting this capability in the hands of users gives a big competitive edge. This also means that BI should allow real-time analysis with simplicity and minimal impact of the system performance.

 

Multiple databases and transaction systems results in information heterogeneity:

  • Syntactic – Same business information retained in different database tables across.
  • Semantic – Same business elements being referred to in different terminology. The result is long report writing cycles and wait time.

Geographical dispersion of databases. Here, the underlying challenge involves limited bandwidth between the various data sources, as well as between data sources and BI users. In case of international locations, time zones can be a big factor as well. BI should solve this problem seamlessly via an integrated network of collaborating subsystems and also make the communication more effective through simple, logical means such as federation of queries and data compression.

 

Complex Analyses and End-User Flexibility

 

Too many reports. Often, tools bind the data sources into each analysis/report. This restricts end users from being able to select the data sources to analyze information from. The end result - a different report variation for each combination (thus exploding the number of distinct reports that needs to be maintained). Data source selection should be external to each analysis and determined by the business user.

 

Cross-functional analysis. This is also true in the case of cross-functional analysis. If a report is bound to a ‘data mart’ or a departmental analysis, the user finds it difficult to expand the analysis across a process. This results in numerous incongruent reports. A single view across the process would make it much easier to measure and manage overall business performance.

 

BI Side Effects – Added Complexity

 

The implementation of BI has matured over time. In the earliest stages, power users equipped with spreadsheets exerted control over the content and display of information.

 

The IT department strictly controlled access to the company’s data. Most requests were handled as one-off reports. The resulting effect was mostly to add work to a small number of low-level workers.

 

As the demand for more information rose, data warehouses were introduced. Data warehouses require specialized software, new database structure and new hardware to house the resulting data. Additional personnel are required to plan, implement and manage this infrastructure. Anybody who accesses the data needed special training. Company growth and business changes required expensive changes to the data warehouse.

 

In the 1990s, organizations attempted to build their BI infrastructure data warehouse elements in a top-down monolithic fashion. These large-scale, enterprise-class projects had trouble delivering value to the business, with studies showing failure rates from 30 percent according to the Meta Group, up to 80 percent.  Inserting BI applications into an organization may necessitate many changes which, of course, increase the complexity of the system.

 

Some BI Side Effects

 

Many BI solutions require additional computer hardware (application and database servers), which may require additional personnel or enhanced skill sets, resulting in increased costs, overhead and, complexity.

 

New software, database management systems and hardware platforms will all require training and support. IT and users must be thoroughly briefed on the current systems as well as given a clear understanding of the desired results of the BI implementation.

 

If any of the training is incomplete or insufficient, it has a cascading impact on the overall implementation. Also, heavy reliance on knowledge and skills means the system becomes very human-dependent.

 

The more parts in the framework, the more upkeep required. It is not uncommon to hear of organizations having dedicated personnel to manage a BI system. While it may not be totally avoided, depending on the number of enterprise users and the volume of data and information, dedicated resources translates into additional cost of ownership.

 

Principles to Managing BI Complexity

 

Databases are becoming faster and more efficient. Processor speed is getting cheaper. This means that there is ample scope to achieve the same end results by leveraging the databases directly rather than building a staging area.

 

Middleware technology has advanced significantly and is capable of managing complex queries. An integrated multithreaded middleware allows federated, parallel queries that utilize each database resource most effectively. For business, this means, real-time information is becoming easily accessible.

 

Business requires reporting, but static reports are not enough to provide answers to business questions. Also, business needs access to enterprise-wide information from the requisite data sources. Managers need to focus on the functional and process performance, not just departmental performance.

 

While most BI vendors have solved the basic data access issues, just being able to access the data from various databases is not enough. Combining the information in an efficient manner resulting in consistent harmonized, easy-to-digest presentation is imperative for success.

 

Business managers need information to analyze business, and their knowledge and skills are focused on measuring and managing business rather than learning new software. It is necessary to ensure that the business users are comfortable with the new means of accessing information.

 

Leverage the data directly from the databases or transaction systems. This helps get to the source of information, thus increasing the level of confidence in the reports. This is critical in today’s environment of compliance and audits.

 

Leverage the native database - SQL and data optimization tools, etc. This way, there is no need to learn new proprietary systems or languages. Also, this ensures reusability of the BI work by other applications.

 

Utilize every piece of hardware in the ‘query and analysis’ process. Leverage the existing user PCs and existing hardware without forcing unnecessary upgrades. With desktops and laptops becoming more and more powerful, their processing power can be leveraged, thus minimizing the need for a heavy middleware server.

 

Make use of off-peak hours to maximize throughput of queries and reports from the existing systems.

 

Do not introduce redundant databases unless absolutely necessary. This includes data warehouses data marts and any redundant stored data. Selective summarization and storage, if at all required, is much more efficient than mass storage and allows a lot of room for growth.

 

Business users need to focus on analyzing and managing business. They don’t need to be report writers. Keep the user layer simple and clean but provide analysis tools to help them answer questions or investigate. Do not expect users to learn database terminology or any technical skills.

 

A business-oriented metadata model is required to achieve this. Also, in multiple database situations where there is syntactic heterogeneity between databases this metadata model is required to normalize the various systems.

 

In situations where there is a business need for cross-functional or process analysis spanning multiple data sets, BI needs to modularize the analysis task and allow each component to be independently specified and incrementally validated. This way as business grows and there are additional areas that need to be analyzed, those analysis modules can be plugged into the metadata without the need to define all the analyses all over again.

 

Semantic heterogeneity can be managed easily by creating a centralized corporate standard master file database that encodes the official customer/item/vendor numbers and then cross-reference that database to each local database.

 

In this paper, I began by exploring the kinds of complexity managers see in business today. I then focused on how BI can be used to help manage these increasingly complex business environments.

 

BI complexity issues are getting a lot of attention since executives are directly noticing the results in the form of longer lead times to get information, higher uncertainty in making decisions and greater administration costs. Often this complexity can be avoided, and businesses can see instant and ongoing benefits if some basic principles are followed: keep the business goals in mind, leverage technology developments, leverage what already works, and keep simple things simple.

 

BI solutions built using those principles greatly enhance user satisfaction, provide immediate ROI and are nimble so they can adapt quickly to business changes and grow as the business grows. In closing, business intelligence is not a cure-all for organizations, but it certainly can help organizations control complexity and manage their businesses better by greatly simplifying the decision-making process without sacrificing any the technical benefits that such solutions offer. 


Dhananjay Joshi is responsible for managing the product lifecycle and evangelizing the products to the market and customers. Joshi has experience leading and managing enterprise resource planning (ERP) and business intlligence implementations for global corporations as well as integration of Exact Business Analytics' business intelligence solution with major enterprise systems.

For more information on related topics, visit the following channels:



Industry Vendors