||View Job Listings|
||Post a job|
Data Strategy Adviser:
A Rational Approach to Chargeback for Data Warehousing, Part 1
||Column published in DMReview.com
February 16, 2006
The Gold Standard for Chargeback
It was really an eye opener for me to have a client describe "chargeback" as his toughest business intelligence (BI) problem. While at first tempted to dismiss the statement as an outlier, further consideration showed this firm's situation to be common, very common. Many enterprises would be able to extend their BI success (and, for example, use more BI products or use them better) were it not for a clumsy chargeback system that artificially inflates costs and thereby discourages use. The situation is so intractable that most firms have given up trying to deal with it. The pain is hidden, denied. But what if there was a way out of the dilemma? This two-part series aims to show the way by developing some alternatives for a rational approach to chargeback for the data warehouse.
Chargeback for the use of computing resources in the mainframe data center is arguably the gold standard for optimizing the use of computing resources. However, the results are less positive when this standard is applied to BI, data warehousing and ad hoc market research environments. Obstacles arise. It does not work. It is too expensive, actually discourages use of the computing resource, and often results in billing disputes and organizational conflicts. Expense, complexity, disputes and conflicts are clear symptoms that the process is broken. What are other models for optimizing use of the resource in the ad hoc decision support ("research") environment typical of BI? In order to conceptualize a different method of chargeback, it will be useful to understand the differences between the traditional data center environment and the exploratory data warehousing (research) one.
Differences Between the Data Center and the BI Environments
The data center environment is fundamentally different from the ad hoc ("exploratory data warehouse") environment. Differences include:
- Work in the data center is well defined in advance. Update the customer master file. Format and route report. Work in the exploration (research) data warehouse is rarely defined in advance. Which customers are good candidates to receive a marketing promotion? What is the product forecast for product ID XYZ for the next quarter?
- Work in the data center runs the business on a day-to-day basis. A customer is on the phone, where is my stuff? Or what is my open inventory? Whereas work in the data warehouse takes a longer view as to what are market trends, strategies for growing top line revenue and strategies for reducing costs.
- Work in the data center is nondiscretionary in comparison with that in the exploratory data warehouse. If you cannot open or update a customer account, then the enterprise is out of business; whereas if you cannot predict market trends, customer profitability or mortgage risk, the enterprise is merely at a competitive disadvantage (not a good thing, to be sure).
- Work in the data center is update intensive, whereas in the data warehouse it is query intensive.
Even though open systems are dramatically less expensive than standard data center ones, numerous accounting and quasi-political issues prevent simply charging a dramatically lower rate for the open system alternatives off of the same system. These include:
- Any cost no matter how small is still greater than zero. No matter how low, chargeback by CPU second (etc.) will discourage use of the BI resource. This is not an issue in the case of transactional systems, because you need to use the resource to perform order entry or account maintenance, or an enterprise will literally be out of business; whereas an enterprise does not need to grow top-line revenue through information productivity or reduce costs through superior forecasting. In the case of business intelligence, the enterprise will merely be at a competitive disadvantage but will still be able to open for business tomorrow morning. Of course, it is obvious such an enterprise will be at a disadvantage even on a quarter-to-quarter basis, but such a disadvantage will creep up relatively quietly unlike the ability to answer the questions of a customer who is on the phone right now. (This is likely happening now due to arcane, bureaucratic rules that have out-lived their usefulness but may not be visible to management until it is too late.)
- You can't fight city hall. Data center chargeback systems have been in place for years. The battles over why the resource costs so much have been fought and refought many times. It has the aura of "you can't fight city hall" about it. If a new and different rate is proposed for open system, the entire framework of chargeback is likely to be called into question. And, while the system is far from perfect, it does have its uses in optimizing the use of the large, expensive mainframe configuration.
- Challenges of instrumentation and data capture. Data center chargeback may present technical obstacles to including open systems, though if it were the right thing to do, such obstacles would have to be engaged as problems to be solved and surmounted. The open system has to be instrumented to return the same level of detailed, granular data by user, application or job that represents utilization in canonical billing format. In the worst case, a parallel billing system would have to be constructed that implemented the same kinds of rules on a different platform.
- Sticker shock. If users see a dramatically lower cost for the BI environment in comparison with the data center cost, then billing disputes will break out due to sticker shock. Do you really want to broach this issue (again)? The data center really is optimized for operational computation (as opposed to BI), and it is not practical to retrain the entire staff to the framework needed for open systems.
Principles for a Simplified Chargeback System
Therefore, because the BI environment is different than the data center environment, the conclusion is that a different method of chargeback is needed for a different kind of environment. What principles should be the basis for formulating and implementing such a method?
Propose a few, simple principles for a rational chargeback system for BI. These include:
In order to avoid methods that discourage use of the business intelligence resource and result in bureaucratic, organizational conflicts, enterprises have focused on utilization as the standard method for metering departments (or the relevant corporate unit) proportionately to the use of the shared business intelligence resource. Utilization of the computing resource is an acceptable measure of effort (how hard are you working?), but is unable to distinguish someone who is working inefficiently from someone who has a lot of work to do and is performing optimally. Utilization does not measure value. This is not a problem with well-defined workloads (as in the data center) that have been optimized over time through extensive tuning (as well as trial and error). Outside of the data center, utilization is a far from ideal measure of value. Even worse, utilization defeats the very purpose of chargeback, which is to optimize the use of the computing resource. Is any way available out of this impasse? Next month this column will look at several alternative methods of representing value that can be used as the basis of a chargeback system for exploratory data warehousing and the business intelligence applications which the data warehouse enables.
- Centralized. This is chargeback for a centralized system that intends to be a shared resource for many departments, users or lines of business.
- Simple (less is more). Avoid a model of chargeback that requires a billing system as complex as the BI system being metered.
- Friendly (or at least not punitive). Avoid a chargeback system that inspires or causes billing or organizational disputes. Avoid a chargeback that discourages use of the resource (i.e., the exploratory data warehouse). For example, levying a cost per query or per cost-based query will discourage queries.
- Value-based. Implement a chargeback that traces the value derived from the system to the line of business deriving benefit from the BI system.
For more information on related topics visit the following related portals...
Business Intelligence (BI).
Lou Agosta, Ph.D., joined IBM WorldWide Business Intelligence Solutions in August 2005 as a BI strategist focusing on competitive dynamics. He is a former industry analyst with Giga Information Group, has served as an enterprise consultant with Greenbrier & Russel and has worked in the trenches as a database administrator in prior careers. His book The Essential Guide to Data Warehousing is published by Prentice Hall. Agosta may be reached at LoAgosta@us.ibm.com.