Portals eNewsletters Web Seminars dataWarehouse.com DM Review Magazine
DM Review | Covering Business Intelligence, Integration & Analytics
   Covering Business Intelligence, Integration & Analytics Advanced Search

View all Portals

Scheduled Events

White Paper Library
Research Papers

View Job Listings
Post a job


DM Review Home
Current Magazine Issue
Magazine Archives
Online Columnists
Ask the Experts
Industry News
Search DM Review

Buyer's Guide
Industry Events Calendar
Monthly Product Guides
Software Demo Lab
Vendor Listings

About Us
Press Releases
Advertising/Media Kit
Magazine Subscriptions
Editorial Calendar
Contact Us
Customer Service

10 Years of Information Quality Advances: What Next?

  Article published in DM Review Magazine
February 2001 Issue
  By Larry English

Ten years ago, a publication called DM Review made its debut. Let's take a moment to reflect on lessons learned in the past 10 years of information management. How far we have come? What challenges remain as we journey into the first decade of the third millennium AD?

The past decade has seen the most dramatic growth in the information age, not just in technology advances, but also in business transformation. A radical transformation in how work is performed brings new economics. This type of radical transformation requires two ingredients: a major and radical technology shift, and a fundamental paradigm shift in principles of how work is performed to exploit the new technology.

The Agricultural Age:
From Hunting and Gathering to Crop Cultivation

The agricultural age came about as the result of new technology (tools) and transforming principles that allowed cultivation of land and domestication of animals. The technologies and principles of hunting and gathering were superceded. A nomadic or semi-nomadic life of searching for and moving to the food gave way to a life of "civilization" with a more permanent location. Work was organized around the land, rather than the natural resources.

As the agricultural age matured, commerce evolved into the cottage industries of crafts. Trades emerged with persons learning skills to craft products with manually operated tools.

The Industrial Age:
From Trade Crafts to Specialization of Labor

The industrial age came about as the result of new principles of work, described by Adam Smith in Wealth of Nations. Specialization of labor became the founding principle of work in the new economy. The breakdown of labor into small sets of tasks allowed for faster production of goods.

Radical productivity advances were a result of the new technologies - power applied to work. James Watt's improvements to the steam engine forever altered how work would be performed in the industrial age. Standardization of component parts and repeatable processes along with Henry Ford's automated assembly line provided automated mass production and new increases in productivity.

Alas, part and parcel of the industrial age was the waste of scrap and rework that came with mass production. Defects along the assembly line were considered natural to the manufacturing process until quality visionaries such as W. Edwards Deming, Joseph Juran, Kaoru Ishikawa, Walter Shewhart and other quality thought- leaders proved that manufacturing scrap and rework did not have to be part of the process. It is possible to improve and control the processes to eliminate the high costs - often 20 to 30 percent of operating revenue - of scrap and rework.

The Japanese transformed the economics of the industrial age by applying quality principles to manufacturing. Freed from the burden of passing on the high costs of manufacturing waste, they competed on an uneven playing field - higher quality products for lower costs.

The quality revolution in manufacturing signaled the maturity of the industrial age. Productivity now meant producing quality goods close to 100 percent of the time rather than 70 to 80 percent of the time - due to scrap and rework - at a fraction of the cost.

The Information Age:
From Dis-integrated Departmental Work to Collaborative Knowledge Work

This last decade saw the 50th anniversary of the information age. From the electromechanical Mark I and Mark II behemoth computers and the vacuum tube UNIVACs of the 1940s to today's parallel processing supercomputers, we have made a quantum leap in processing speeds. From the Mark I's three addition operations per second to trillions of instructions per second, we can perform work far faster than ever imagined at the outset of the information age - and can create data errors just as fast.

Information technology is not about increasing productivity of industrial-age work; it is about transforming how work is done altogether. Without understanding of the principles of the information age, organizations will automate the wrong work, failing to exploit the tools of the new economy.

At the time of the birth of DM Review, a formal concept of information quality improvement (applying quality principles of process improvement and control to information as the product of business and manufacturing processes) was just emerging.

Pre- 1991:
Information Quality Uncertainty

Prior to 1991, no books on information quality existed. There was no formal methodology applying total quality management (TQM) principles to information, and only a handful of software companies had products that addressed information quality in any way.

The Birth of Information Quality Management

In January of 1991, Mark Hansen's master's thesis, "Zero Defect Data: Tackling the Corporate Data Quality Problem" (MIT Sloan School of Management), was published. Also in 1991,QDB Solutions was formed. QDB created a software product for assessing data validity that produced Pareto diagrams of invalid data by type along with histories of assessments similar to control charts. Dr. Richard Wang established the Total Data Quality Management (TDQM) program at MIT, which he has co-chaired with Stuart Madnick; Tom Redman was writing his first book on data quality (published in 1992); and I discovered W. Edwards Deming's 14 Points of Quality and began applying them in a formalized methodology called TQdM (Total Quality data Management).

From Uncertainty to Awakening

The first half of the 1990s saw an increase in information quality problems, spurred by the increase in islands of automation as the new technologies of client server and data warehousing proliferated redundant databases across many platforms. This was accompanied by the emergence of a number of data cleansing and data integration products. Most of the information quality product development focused on reactive, corrective maintenance. These cleansing products automated some of the processes of information scrap and rework.

From Awakening to Enlightenment

The past five years have seen not just an increase in information quality problems, but an accelerating increase in information quality problems. This is caused by an accelerating increase in information technologies and an accelerating loss of control in the information management processes.

At the same time, awareness and maturity in information quality processes have grown. Through eight information quality conferences I have organized and chaired in the U.S. and Europe, I have noticed that information quality maturity in pioneering organizations has grown. There were just a handful of information quality products before 1991; now there are approximately 200 different information quality products in various categories of information quality assessment, analysis, management, corrective maintenance and defect prevention. Significantly, there has been a maturating in information quality tools to include or increase defect prevention capabilities. By invoking libraries or routines to apply information quality rules, applications that create data can prevent errors at the source and eliminate process failure and subsequent data correction.

The second half of the 1990s saw early leading organizations adopt a proactive defect- prevention mind-set with a culture of customer satisfaction in their information products. With it, the enterprises focused on information quality reaped the rewards of decreased costs of information scrap and rework and increased customer satisfaction and opportunity gains.

The Next Decade of Information Quality Management
Scenario 1: Threat of Enterprise Failure

Organizations that do not apply information age principles will continue to sub-optimize information technologies. Those organizations run the risk of being strangled in a mire of spaghetti systems and increased costs of information scrap and rework. Repeatedly fixing the same problems increases the risk of enterprise failure.

History repeats itself. We are seeing spectacular failures occur just as fast - if not faster - than the technology changes. For example:

  • The incredible rise and fall of the e-empire (tack an "e-" to the beginning of your name, go on the Web and make lots of money) should prove to the world that Web technology as the solution is a false holy grail. To be successful in e-business, you must have a business plan driven by meeting customer expectations.
  • Object orientation was the technology that was supposed to replace relational database and traditional programming languages. Did it? No. A technology built around the notion of reuse can be used to produce objects with minimal reuse. One pilot object project I reviewed had an object class naming standard where the class name began with the first two letters of the application ID! The irony was that the goal was to achieve a tenfold increase in object reuse; the pilot project actually accomplished a reuse rate of only 10 percent.
  • Data warehousing was probably the biggest hope for business management to finally get the information they need. Success rate? Maybe one-half of data warehouse projects meet some of their objectives and satisfy some immediate information customers. However, according to one of my associates, only as few as eight percent of data warehouses are considered long-term successes when measured by increase in information customers (market share) and increase in data brought into the data warehouse (new product development).
  • Application development. In 1998, the Standish Group surveyed approximately 7,000 IT projects and discovered only 26 percent of them were considered successes, compared with 28 percent that failed while 46 percent had significant cost overruns. With that as a benchmark, one would expect 1999 rates to have no place to go but up. So, what did the 1999 Standish Group survey reveal? The success rate actually decreased, from a 1998 success rate of 26 percent to a miserable 17 percent. Failures grew 18 percent to a total of 34 percent, and 49 percent of the projects had significant cost overruns.
  • Repositories were created to help IT manage and control data and applications. Why was it, however, that when the Y2K problem loomed, the teams had to scurry around to document, most for the first time, their inventories of applications and files? And, why are many of those inventories now out of date and no longer maintained?
  • Database management systems were supposedly the solution for true data integration through data sharing from common shareable databases rather than through data passing. What happened? Most organizations kept developing applications and DBMS files using the same project-by-project development methods they had used in the past. The result was application databases, using the expensive DBMS "data sharing" technology as an expensive access method!

The conclusion here should be obvious: The mere use of any information technology will not succeed without the understanding of the paradigm of the information age and the principles of information management and process transformation. This conclusion calls us to the find out why so many information systems have failed. We must understand the principles of the information age and transform how we manage the enterprise and information technology.

Scenario 2: Promise of Business Performance Excellence

The understanding and application of quality principles led to the maturing of the industrial age. So also will the understanding and application of information quality principles lead to the maturing - and success - of organizations in the information age. The principles are clear:

  • Manage processes across the business value chain, not down the function.
  • Manage information as a strategic business resource, applying sound management principles (planning, organizing and staffing, leading and directing, controlling and exploiting).
  • Manage information as a product, applying sound quality management principles to information:
    a. Understand the "information customer's" quality requirements, and manage information quality to meet customer expectations.
    b. Assess defective data at its source.
    c. Apply the Plan-Do-Check-Act method to identify the root cause of data defects and improve the process to eliminate recurrence.
    d. Transform the culture through training and education.
    e. Transform the performance measures that reward quotas and speed because these measures create defective data and cost far more in information scrap and rework and process failure.
  • Implement accountability in business management for business information.
  • Implement accountability in information systems management for quality of information systems products - not just "on time, within budget" goals.

The information age organization manages processes as value chains horizontally across business functions to optimize the entire value chain, not just a functional component of the value chain. All functions and organization units are interdependent. Functional activities and the information products produced must be managed to meet the downstream information stakeholders' requirements. If data produced by one function meets only that function's needs but causes downstream functions to fail, is that data quality data or not?

The question is not whether, but when. Will the maturity of the information age come in the first decade of the 21st century or will more spectacular enterprise failures be required?

What do you think? Let me know at Larry.English@infoimpact.com or on the IQ Forum under IQ Resources at www.information-quality.com.


For more information on related topics visit the following related portals...
Data Quality.

Larry P. English is president and principal of INFORMATION IMPACT International, Inc., Brentwood, Tennessee, and the author of the widely acclaimed book, Improving Data Warehouse and Business Information Quality: Methods for Reducing Costs and Increasing Profits. English is cofounder of the International Association for Information and Data Quality (www.iaidq.org). English is an internationally recognized speaker, teacher, consultant and author and may be reached at larry.english@infoimpact.com or through his Web site at www.infoimpact.com. For more on how to improve your IQ principles and techniques, and prevent your organization from wasting millions in information scrap and rework, join the IAIDQ (visit www.iaidq.org).

Solutions Marketplace
Provided by IndustryBrains

Data Validation Tools: FREE Trial
Protect against fraud, waste and excess marketing costs by cleaning your customer database of inaccurate, incomplete or undeliverable addresses. Add on phone check, name parsing and geo-coding as needed. FREE trial of Data Quality dev tools here.

Recover SQL Server or Exchange in minutes
FREE WHITE PAPER. Recover SQL Server, Exchange or NTFS data within minutes with TimeSpring?s continuous data protection (CDP) software. No protection gaps, no scheduling requirements, no backup related slowdowns and no backup windows to manage.

Speed Databases 2500% - World's Fastest Storage
Faster databases support more concurrent users and handle more simultaneous transactions. Register for FREE whitepaper, Increase Application Performance With Solid State Disk. Texas Memory Systems - makers of the World's Fastest Storage

Design Databases with ER/Studio: Free Trial
ER/Studio delivers next-generation data modeling. Multiple, distinct physical models based on a single logical model give you the tools you need to manage complex database environments and critical metadata in an intuitive user interface.

Free EII Buyer's Guide
Understand EII - Trends. Tech. Apps. Calculate ROI. Download Now.

Click here to advertise in this space

View Full Issue View Full Magazine Issue
E-mail This Article E-Mail This Article
Printer Friendly Version Printer-Friendly Version
Related Content Related Content
Request Reprints Request Reprints
Site Map Terms of Use Privacy Policy
SourceMedia (c) 2006 DM Review and SourceMedia, Inc. All rights reserved.
SourceMedia is an Investcorp company.
Use, duplication, or sale of this service, or data contained herein, is strictly prohibited.