FREE DM Review Site Registration!
Sign-up today and access DM Review on the Web!

Your FREE registration entitles you to:
FREE email newsletters

FREE access to all DM Review content

FREE access to web seminars, resource portals, our white paper library and more!

   

The Data Warehouse Satisfaction Survey, Part 1: The Number One Complaint About Data Warehousing

The IBM Data Warehousing Satisfaction Survey (2007) consisted of an invitation to some 200 end-user enterprises to participate in an anonymous, Web-based survey about data warehousing architecture, latency, size and related business issues. Invitations were sent to enterprises regardless of the data warehousing platforms they were using, and respondents included the complete spectrum of what is in the market at this time, including IBM, Microsoft, Netezza, Oracle and Teradata platforms. The emphasis was on surfacing trends that apply regardless of the specific data warehousing platform. Here is a look at some of the initial results of the survey.1

Data Warehousing is not a Paradigm Shift - Innovations Continue

The IBM Data Warehousing Satisfaction Survey shows that more than 56 percent of data warehouses have been in production for over six years (see Table 1). Only 19 percent have been in production for three years or less. In short, data warehousing is a mature approach to delivering information for business intelligence and does not itself represent a paradigm shift. Rather, data warehousing is the foundation for step-by-step incremental progress in data management across an impressive and varying array of fundamental business applications as will discussed in more detail in this report. The conclusion is that data warehousing is a sustaining force in data management and application development, and new innovations and advances continue to drive evolution of the overall market.

Table 1

With that in mind, clients should build for the long term, since data warehouses tend to live long lives. Data warehousing is not a technology one-night stand; it is a long-term commitment. Of course, this is consistent with innovations in data warehousing technology such as XML for unstructured data, advanced applications in data mining, design advisors to help the database administrator work smarter or dynamic warehousing approaches to enable extended use of business intelligence. This is also relevant to installing an appliance or an appliance-like data mart - just understand that the enterprise will end up living with it for a long time.

Success of Data Warehousing Implementations Growing

A growing portion of enterprises (56 percent) report that they are "very successful" with data warehousing in light of their own objectives (see Table 2). Unlike the early days of data warehousing in the 1990s when a majority of projects did not satisfy the minimal essential set of requirements, today projects to build and operate a data warehouse are more likely to succeed. This is not to say that all challenges have been eliminated - those organizations at a relatively low level in the data warehousing capability maturity model still struggle with meeting deadlines, handling service level agreements and satisfying end-user requirements. Even allowing for a certain amount of rosy self-reporting - we have never met a project team that would volunteer information to a survey, no matter how anonymous, that the project was a failure - practitioners are becoming increasingly satisfied with the results of data warehousing. No one acknowledges having produced shelfware. That means enterprises are getting value and that the staff are hanging onto their jobs. As indicated, this is reflected in Table 2 where more than half of respondents reported that their data warehousing is "very successful in light of the objectives." Still, it should be no surprise that in the real world 43 percent of respondents acknowledge room for improvement. Below we will "drill down" on the key issues organizations continue to face.

Table 2

Number One Complaint: My Data Warehouse Lacks Data

In spite of all the good news about data warehousing, much opportunity for improvement remains. Table 3 indicates organizations' responses on what continues to not work satisfactorily. A wide variety of business and technical challenges still confront enterprises operating data warehouses. As noted in Table 3, the number one source of dissatisfaction with data warehouses in production, weighing in at about 51 percent, is a lack of needed data. In other words, these data warehouses lack data. This is a shocking result - data warehouses that lack data! Don't let this happen to your system. What could this possibly mean - that my data warehouse lacks data? In a few selected follow-up conversations between survey respondents and the survey team, what came out was that specific kinds of data, especially customer data but also product data and other dimensions, were missing for a wide variety of reasons. This is reflected in the next most frequent complaint of "insufficient or inadequate master data," which some 41 percent of respondents asserted. It was also found that many organizations are not leveraging their information as effectively as they could to have a direct impact on the business, as can be seen in the 24 percent of respondents that indicated their warehouse was not satisfying all or most of the business requirements. Other sources of trouble included poor price performance, too many silos and lack of consistent results.

Table 3 (Percentages will sum to more than 100 percent due to allowing multiple responses.)

One respondent suggested that her data warehouse was like a theatrical facade on a Hollywood movie set. When you looked behind the building, it was just a one dimensional, cardboard front - lacking data - and "you are in a really mean neighborhood." This professional had been hired (in effect) to clean up the mess by governance that said "everyone does their own thing." The result was growing coordination costs and a cost profile on a slope of diminishing returns, running faster and faster to keep up with maintenance.

In conclusion, the number one complaint about data warehouses in production is "the data warehouse lacks data." That is, customer data or product data or pricing data or some minimal, essential data source is missing, unavailable or otherwise an issue. This lines up with complaints about lack of master data, which is also a top negative, though data warehouses were among the first systems to conceptualize the design and implementation of master data dimensions for decision support. This result leads to the prediction that master data management software providers will renew their commitment to supporting MDM for both business intelligence as well as transactional systems. In the meantime, data warehouses will continue to "wag the dog" as the back-end data warehouse requires upstream transactional systems to clean up their MDM in order to support increasingly strategic decision support initiatives.

1. Editor's note: Phase 2 of the IBM Data Warehousing Satisfaction Survey is now live! Interested readers are invited to participate by clicking on the following URL and spending 20 minutes answering some 23 questions in this anonymous, Web-based survey. https://www14.software.ibm.com/iwm/web/swg-dwss/entry.shtml.


Lou Agosta is an independent industry analyst in data warehousing. A former industry analyst at Giga Information Group, Agosta has published extensively on industry trends in data warehousing, data mining and data quality. He can be reached at LAgosta@acm.org.

Marc Andrews is the program director, IBM Data Warehousing. He speaks and writes extensively on IBM's dynamic data warehousing initiative and related topics.

Mark Ritzmann is a leader in the IBM World Wide Business Intelligence Group. Ritzmann has written and spoken widely on issues in data warehousing and business intelligence.

For more information on related topics, visit the following channels:


Industry Vendors