Portals eNewsletters Web Seminars dataWarehouse.com DM Review Magazine
DM Review | Covering Business Intelligence, Integration & Analytics
   Covering Business Intelligence, Integration & Analytics Advanced Search

Resource Portals
Analytic Applications
Business Intelligence
Business Performance Management
Data Integration
Data Quality
Data Warehousing Basics
More Portals...


Information Center
DM Review Home
Conference & Expo
Web Seminars & Archives
Current Magazine Issue
Magazine Archives
Online Columnists
Ask the Experts
Industry News
Search DM Review

General Resources
Industry Events Calendar
Vendor Listings
White Paper Library
Software Demo Lab

General Resources
About Us
Press Releases
Media Kit
Magazine Subscriptions
Editorial Calendar
Contact Us
Customer Service

Data Warehousing Lessons Learned:
The Value of Performing a Custom Benchmark

  Column published in DMReview.com
April 1, 2002
  By Lou Agosta

A prospective buyer of data warehouse technology can learn a great deal from performing a benchmark specific to his or her firm's data center and operations. If an enterprise is contemplating spending millions of dollars with database warehousing and hardware vendors, it would be wise to perform a custom benchmark based on a firm's specific data profile. It provides an opportunity to verify that the investment will satisfy the requirements as well as an opportunity to negotiate with the vendor for concessions. However, if a firm already has significant experience with a vendor (for example, it already operates with the vendor's database or hardware), a benchmark is less useful or necessary, though by no means should it be ruled out on principle.

If a vendor thinks you are a serious buyer who may spend a million dollars, the vendor will often perform the benchmark at no extra cost (or minimal cost) to the prospect enterprise. Thus, the cost to a specific enterprise in terms of dollars changing hands in a transaction may be zero, provided the enterprise has built a credible case that it wishes to buy and provided the vendor succeeds in meeting the performance requirements.

Obviously, this is a delicate negotiation that requires planning and effort on the part of the buying enterprise ? so there is a cost in time and effort to the prospective buyer; however, it is not a transaction cost in the market.

Along the same lines, in order to derive value from the benchmark, the prospect must be prepared to devote time and effort to defining the performance requirements, selecting and delivering the relevant data, and participating in the process. That is why a distinction is made between a transaction cost in the market (which is often zero) and a cost to the prospect in terms of staff time and effort (which may be significant).

To appreciate the effort involved and the kinds of questions to ask when performing a custom benchmark, visit the Transaction Processing Performance Council Web site (www.tpc.org). Look at the full disclosure report (FDR) for the public benchmarks at the 300GB, 1TB or 3TB volume points. What are your SQL queries? How much data is involved? What is your data model? Those who are interested in performing a custom benchmark must be ready to answer these questions. Before proceeding with negotiations to perform a benchmark, it is useful for the prospective client to have an inventory of its own data assets (data profile, growth prospects, applications) and an understanding of its performance requirements. If you are the prospect and have done your homework in these areas, then proceed.

Another thing to consider is that the benchmarking center of any given database or hardware vendor has the capacity to perform a couple of dozen benchmarks a year for individual prospective clients at the rate of one or two benchmarks every one or two weeks. Naturally, the larger vendors have more bandwidth, but it is not infinite. Therefore, the vendor must decide, largely based on input from the sales team, which two or three dozen major clients a year would make this effort worthwhile and result in a significant win (i.e., sale). That is what it looks like from the vendor's point of view.

Some vendors, such as DB2 (IBM) and Teradata (NCR), are highly competitive with one another across hardware, software or both. Including them on a short list would increase a firm's chances that the vendors would be willing to perform a benchmark at no incremental cost to the buying enterprise (other than the staff time and management effort). In another twist, if an end-user firm really wants to perform a benchmark, regardless of purchase decision, some vendors would be willing to rent excess capacity if it is available for that purpose. Finally, if the vendor believes you are benchmarking with its competitor, then it will be more likely to compete aggressively for the business.

As indicated, substantial public information is available at the TPC Web site under the TPC-H category. TPC-H represents the decision support benchmarks - "H" stands for "ad hoc" query.

In addition, the database or hardware vendor should be willing to provide the names and contact information for several clients who have visited its benchmark laboratory and allow you to talk to them confidentially. They will likely tell you that it is a worthwhile effort, even though it requires considerable effort on your part. A recurring theme ? and data warehousing lesson learned ? is that benchmarking is one of those tasks where you get out of it in proportion to what you put into it.


For more information on related topics visit the following related portals...
Business Intelligence, Enterprise Intelligence, Benchmarking/Best Practices and Metrics/KPI/BSC.

Lou Agosta is the lead industry analyst at Forrester Research, Inc. in data warehousing, data quality and predictive analytics (data mining), and the author of The Essential Guide to Data Warehousing (Prentice Hall PTR, 2000). Please send comments or questions to lagosta@acm.org.



Solutions Marketplace
Provided by IndustryBrains

Enabling the Dynamic Enterprise with CommerceQuest
CommerceQuest offers a complete set of scalable enterprise business process management (BPM) software products and business solutions. Click here to learn more and download free white papers.

Bowne Global - The Language Services You Need!
We are the leading provider of translation, technical writing, interpretation services & more! We enable you to deliver locally relevant & culturally connected products, services & communications anywhere in the world! Request more information today!

Integration Simplified: More Value from IT Efforts
Free from BEA: Guide to Unified Application and Business Integration. Learn how to speed and simplify integration initiatives, reap business value more quickly, and make IT more responsive to real-world needs. Free white papers, case studies, more.

Online CRM solutions from Salesforce.com
Online Customer Relationship Management solutions - sales force automation, customer service and support, and marketing automation. All this and no software! Designed for rapid deployment and adoption. Free 30-day Trial.

Online Courses in Statistics
3-4 week courses in data mining, market segmentation, survey stats, intro stats, regression, logistic regression, time series forecasting... Interact with instructors - leading experts. No set hours - go online at your convenience. 10 hrs/week.

Click here to advertise in this space

E-mail This Column E-Mail This Column
Printer Friendly Version Printer-Friendly Version
Related Content Related Content
Request Reprints Request Reprints
Site Map Terms of Use Privacy Policy

Thomson Media

2005 The Thomson Corporation and DMReview.com. All rights reserved.
Use, duplication, or sale of this service, or data contained herein, is strictly prohibited.