Portals eNewsletters Web Seminars dataWarehouse.com DM Review Magazine
DM Review | Covering Business Intelligence, Integration & Analytics
   Covering Business Intelligence, Integration & Analytics Advanced Search

View all Portals

Scheduled Events

White Paper Library
Research Papers

View Job Listings
Post a job


DM Review Home
Current Magazine Issue
Magazine Archives
Online Columnists
Ask the Experts
Industry News
Search DM Review

Buyer's Guide
Industry Events Calendar
Monthly Product Guides
Software Demo Lab
Vendor Listings

About Us
Press Releases
Advertising/Media Kit
Magazine Subscriptions
Editorial Calendar
Contact Us
Customer Service

Volume Analytics:
Beware of Bad Metrics

online columnist Guy Creese     Column published in DMReview.com
June 16, 2005
  By Guy Creese

Given the daily mix of meetings, projects and crises in a BI practitioner's world, it is not always easy to step back and look at the bigger picture. However, it's critical to do so from time to time - a bit of questioning and rethinking can go a long way toward making sure a company continues to analyze itself and its world correctly.

One area that needs conscious rethinking from time to time is the quality of your company's metrics. Typically, bad metrics creep into spreadsheets and corporate dashboards in an insidious fashion. It isn't that anyone consciously proposes metrics that will mislead the company. Instead, sometimes comes up with a pet count or a ratio for the want of anything better; it gets used in a couple of spreadsheets; and then all of a sudden a well-meaning person inserts it into the corporate dashboard.

The evil of bad metrics is that they can turn a company against itself - making it waste time responding to the wrong problem or encouraging it to mistreat its customers. Therefore, following are some pointers to help you root out bad metrics.

Bad Metrics #1: Inaccurate Metrics

The first problematic metrics are those that are the correct ones to follow but have been calculated incorrectly. For example, companies typically use Web site traffic as an indicator of customer interest: e.g., the number of daily hits and the number of page views, among other things. However, some hits are not even affiliated with customer interest - for example, hits created when Google and Yahoo! crawlers index the site. Therefore, it is worthwhile to ask whether the metric accurately reflects the business purpose. In this case, it's easy to turn an inaccurate metric into an accurate one - merely subtract out the hits from Web crawlers.

To give another online example, several years ago a well-known entertainment company was crowing internally about its ability to capture the e-mails of interested customers. The company made users register to play an online game, and a name and an e-mail address were the price of admission. However, rather than using the e-mail address as a unique key, the company demanded unique logon IDs. Once the number of e-mail addresses got above five million, the company decided to use them in an e-mail marketing campaign and found to its dismay that about half of them were duplicates. It turns out that customers didn't come back to the site very frequently and often forgot their password. Rather than go through the hassle of having the password sent to them, they just created a new logon. Again, an easy metric to fix: remove the e-mail duplicates.

Bad Metrics #2: Inappropriate Metrics

Another problem is metrics that are calculated correctly but lead to the wrong behavior. To take a specific example, customer support metrics that track response time often fall into this category. If a customer support representative (CSR) is measured on how quickly he or she picks up the phone, you can guarantee the CSR will do that. Of course, CSR may then put the customer on hold for three minutes, but hey, the measured initial pickup was fast.

Another example from my past history as a CSR was "the lunchtime subterfuge." A number of years ago I supported complex client/server software. If you couldn't solve the problem instantly, it typically took hours to debug and days, if not weeks, to resolve. For those longer-term problems, we were under orders to call customers back within 24 hours to get additional information and give them an update. To meet that metric, we would check the customer's area code and call when the customer was at lunch - the metric police were happy and we avoided talking to the customer - which gave us more time to take and close the easy questions. Since we were graded on how many calls we closed without reference to their difficulty - according to the company, closing 12 easy calls was twice as good as closing six difficult ones - we made sure we cherry-picked to stay employed.

These incorrect metrics are harder to discover, as they're accurate from a computational point of view - but dysfunctional from a business point of view. The best way to root out these problems is to ask the affected workers if they reward the correct behavior. My experience is that they'll rant and rave about the stupidity of the metrics - and then offer well-thought-out suggestions for better ones. Often, no one had ever asked them before if the emperor was wearing any clothes.

An Example of a Good Metric

If the above examples are metrics gone bad, what's a good metric? I think Amazon.com uses a terrific one. Amazon predicts from minute to minute (based on past history, season, time of day, etc.) what its sales should have been within the past half hour. If the actual figure dips below this calculated threshold at any time, pagers automatically go off and major Amazon.com movers and shakers convene in a conference room to figure out what's gone wrong.

What's great about this metric is two things. First, it's what I would call a "safety net" metric. Amazon is sure that some eventuality will happen that the company has not foreseen. Rather than try and monitor everything known to man - although they certainly monitor a wide variety of systems and metrics - they've created a metric that will capture the unknown. As one employee says, "If our pagers go off, we know something is wrong. We may not know what - maybe some DNS servers went down, maybe we're under a denial of service attack, maybe the checkout process got screwed up - but at least we know we need to investigate it and fix it."

Second, they've allied the metric with action, by connecting it to their pagers. They aren't going to miss it because someone forgot to look at a dashboard. Instead, they know exactly what to do: convene and act.

Drill Down on the Meanings, Not Just the Numbers

Armed with these examples, comb through your metrics and double-check that they're serving you rather than misleading you. Talk to the folks in the trenches, and ask if the metrics that they're being measured against are the right metrics for the company at large. In other words, drilling down on the meanings of the numbers can be just as important as drilling down on the numbers themselves.


For more information on related topics visit the following related portals...
Metrics/KPI/BSC and Web Analytics.

Guy Creese is an analyst with the Burton Group, covering content management and search. Creese has worked in the high tech industry for 25 years, at both Fortune 500 companies and small startups, in positions ranging from programmer to product manager to customer support engineer.  He can be reached at gcreese@burtongroup.com.

Solutions Marketplace
Provided by IndustryBrains

DOVICO Time & Project Tracking Software
Award Winning Project Tracking And Costing Software Application!

Continuous Data Protection - recover it all fast!
TimeSpring FREE WHITE PAPER - click here. Continuously protect Exchange, SQL Server and NTFS with TimeData CDP software. Easiest to use, non disruptive, affordable. "Infinite" choice of restore points.

Data Mining: Levels I, II & III
Learn how experts build and deploy predictive models by attending The Modeling Agency's vendor-neutral courses. Leverage valuable information hidden within your data through predictive analytics. Click through to view upcoming events.

Rosette Linguistics Platform
Basis Technology utilizes powerful techniques to provide software solutions for extracting meaningful intelligence from unstructured text in Asian, European and Middle Eastern languages.

Website tracking, web statistics and analytics
Watch your visitors in real time as they browse your site. Website statistics provide insight. Understanding website traffic and visitor clickstream behavior is crucial to managing a website on a daily basis. Real-Time reporting. 4 week free trial.

Click here to advertise in this space

E-mail This Column E-Mail This Column
Printer Friendly Version Printer-Friendly Version
Related Content Related Content
Request Reprints Request Reprints
Site Map Terms of Use Privacy Policy
SourceMedia (c) 2006 DM Review and SourceMedia, Inc. All rights reserved.
SourceMedia is an Investcorp company.
Use, duplication, or sale of this service, or data contained herein, is strictly prohibited.