FREE DM Review Site Registration!
Sign-up today and access DM Review on the Web!

Your FREE registration entitles you to:
FREE email newsletters

FREE access to all DM Review content

FREE access to web seminars, resource portals, our white paper library and more!

   

What metrics should use to measure performace of an outsurced data warehouse?

Q: If you have a technology company that manages your data warehouse as an ASP, what metrics would you come up with to measure performance?

Sid Adelman's Answer:

A question comes to mind about your contract with the ASP. Were any performance measurements included in your agreement? I'd use the following metrics:

  1. The quality and the currency of the software (ETL, DBMS, BI, data quality, metadata) provided by the ASP. Are the products main line and is the ASP keeping current with new releases?
  2. Availability. This would be the percentage of time the system was available to you (not just if the lights were on at the ASP) during your scheduled hours and days (24x7, 18x6), which I assume is in your contract. You will also want to have a service level agreement (SLA) for maximum downtime.
  3. Response time for benchmark queries. These queries should include simple, medium and complex resource requirement samples. Having a response time SLA for all queries makes no senses since the variability of the queries in terms of resources will probably be very high.
  4. How quickly the ASP responds to new requests.
  5. How well the ASP understands your industry and your vision (subjective but important).
  6. How quickly the ASP responds to problems.
  7. The reasonableness of pricing when you have requirements not included in your contract.
  8. Level of support of the end users (unless you are the support). Be sure to measure the users' perception of the quality of support.
  9. Do your people like working with the ASP? Again, this is subjective but very important.
  10. How well does the ASP provide security and privacy?
  11. Does the ASP keep you informed about the capabilities of their service as well as problems? You want information on problems and how they are being resolved.
  12. Lastly, you want metrics from the ASP on usage, response time and availability.

Sid Adelman is a principal in Sid Adelman & Associates, an organization specializing in planning and implementing data warehouses, in data warehouse and BI assessments, and in establishing effective data architectures and strategies. He is a regular speaker at DW conferences. Adelman chairs the "Ask the Experts" column on www.dmreview.com. He is a frequent contributor to journals that focus on data warehousing. He co-authored Data Warehouse Project Management and is the principal author on Impossible Data Warehouse Situations with Solutions from the Experts and Data Strategy. He can be reached at (818) 783-9634 or visit his Web site at www.sidadelman.com.

For more information on related topics, visit the following channels:



Industry Vendors