|Sign-Up for Free Exclusive Services:||Portals|||||eNewsletters|||||Web Seminars|||||dataWarehouse.com|||||DM Review Magazine|
|Covering Business Intelligence, Integration & Analytics||Advanced Search|
10 Years of Information Quality Advances: What Next?
Ten years ago, a publication called DM Review made its debut. Let's take a moment to reflect on lessons learned in the past 10 years of information management. How far we have come? What challenges remain as we journey into the first decade of the third millennium AD?
The past decade has seen the most dramatic growth in the information age, not just in technology advances, but also in business transformation. A radical transformation in how work is performed brings new economics. This type of radical transformation requires two ingredients: a major and radical technology shift, and a fundamental paradigm shift in principles of how work is performed to exploit the new technology.
The Agricultural Age:
The agricultural age came about as the result of new technology (tools) and transforming principles that allowed cultivation of land and domestication of animals. The technologies and principles of hunting and gathering were superceded. A nomadic or semi-nomadic life of searching for and moving to the food gave way to a life of "civilization" with a more permanent location. Work was organized around the land, rather than the natural resources.
As the agricultural age matured, commerce evolved into the cottage industries of crafts. Trades emerged with persons learning skills to craft products with manually operated tools.
The Industrial Age:
The industrial age came about as the result of new principles of work, described by Adam Smith in Wealth of Nations. Specialization of labor became the founding principle of work in the new economy. The breakdown of labor into small sets of tasks allowed for faster production of goods.
Radical productivity advances were a result of the new technologies - power applied to work. James Watt's improvements to the steam engine forever altered how work would be performed in the industrial age. Standardization of component parts and repeatable processes along with Henry Ford's automated assembly line provided automated mass production and new increases in productivity.
Alas, part and parcel of the industrial age was the waste of scrap and rework that came with mass production. Defects along the assembly line were considered natural to the manufacturing process until quality visionaries such as W. Edwards Deming, Joseph Juran, Kaoru Ishikawa, Walter Shewhart and other quality thought- leaders proved that manufacturing scrap and rework did not have to be part of the process. It is possible to improve and control the processes to eliminate the high costs - often 20 to 30 percent of operating revenue - of scrap and rework.
The Japanese transformed the economics of the industrial age by applying quality principles to manufacturing. Freed from the burden of passing on the high costs of manufacturing waste, they competed on an uneven playing field - higher quality products for lower costs.
The quality revolution in manufacturing signaled the maturity of the industrial age. Productivity now meant producing quality goods close to 100 percent of the time rather than 70 to 80 percent of the time - due to scrap and rework - at a fraction of the cost.
The Information Age:
This last decade saw the 50th anniversary of the information age. From the electromechanical Mark I and Mark II behemoth computers and the vacuum tube UNIVACs of the 1940s to today's parallel processing supercomputers, we have made a quantum leap in processing speeds. From the Mark I's three addition operations per second to trillions of instructions per second, we can perform work far faster than ever imagined at the outset of the information age - and can create data errors just as fast.
Information technology is not about increasing productivity of industrial-age work; it is about transforming how work is done altogether. Without understanding of the principles of the information age, organizations will automate the wrong work, failing to exploit the tools of the new economy.
At the time of the birth of DM Review, a formal concept of information quality improvement (applying quality principles of process improvement and control to information as the product of business and manufacturing processes) was just emerging.
Prior to 1991, no books on information quality existed. There was no formal methodology applying total quality management (TQM) principles to information, and only a handful of software companies had products that addressed information quality in any way.
In January of 1991, Mark Hansen's master's thesis, "Zero Defect Data: Tackling the Corporate Data Quality Problem" (MIT Sloan School of Management), was published. Also in 1991,QDB Solutions was formed. QDB created a software product for assessing data validity that produced Pareto diagrams of invalid data by type along with histories of assessments similar to control charts. Dr. Richard Wang established the Total Data Quality Management (TDQM) program at MIT, which he has co-chaired with Stuart Madnick; Tom Redman was writing his first book on data quality (published in 1992); and I discovered W. Edwards Deming's 14 Points of Quality and began applying them in a formalized methodology called TQdM (Total Quality data Management).
The first half of the 1990s saw an increase in information quality problems, spurred by the increase in islands of automation as the new technologies of client server and data warehousing proliferated redundant databases across many platforms. This was accompanied by the emergence of a number of data cleansing and data integration products. Most of the information quality product development focused on reactive, corrective maintenance. These cleansing products automated some of the processes of information scrap and rework.
The past five years have seen not just an increase in information quality problems, but an accelerating increase in information quality problems. This is caused by an accelerating increase in information technologies and an accelerating loss of control in the information management processes.
At the same time, awareness and maturity in information quality processes have grown. Through eight information quality conferences I have organized and chaired in the U.S. and Europe, I have noticed that information quality maturity in pioneering organizations has grown. There were just a handful of information quality products before 1991; now there are approximately 200 different information quality products in various categories of information quality assessment, analysis, management, corrective maintenance and defect prevention. Significantly, there has been a maturating in information quality tools to include or increase defect prevention capabilities. By invoking libraries or routines to apply information quality rules, applications that create data can prevent errors at the source and eliminate process failure and subsequent data correction.
The second half of the 1990s saw early leading organizations adopt a proactive defect- prevention mind-set with a culture of customer satisfaction in their information products. With it, the enterprises focused on information quality reaped the rewards of decreased costs of information scrap and rework and increased customer satisfaction and opportunity gains.
The Next Decade of Information Quality Management
Organizations that do not apply information age principles will continue to sub-optimize information technologies. Those organizations run the risk of being strangled in a mire of spaghetti systems and increased costs of information scrap and rework. Repeatedly fixing the same problems increases the risk of enterprise failure.
History repeats itself. We are seeing spectacular failures occur just as fast - if not faster - than the technology changes. For example:
The conclusion here should be obvious: The mere use of any information technology will not succeed without the understanding of the paradigm of the information age and the principles of information management and process transformation. This conclusion calls us to the find out why so many information systems have failed. We must understand the principles of the information age and transform how we manage the enterprise and information technology.
Scenario 2: Promise of Business Performance Excellence
The understanding and application of quality principles led to the maturing of the industrial age. So also will the understanding and application of information quality principles lead to the maturing - and success - of organizations in the information age. The principles are clear:
The information age organization manages processes as value chains horizontally across business functions to optimize the entire value chain, not just a functional component of the value chain. All functions and organization units are interdependent. Functional activities and the information products produced must be managed to meet the downstream information stakeholders' requirements. If data produced by one function meets only that function's needs but causes downstream functions to fail, is that data quality data or not?
The question is not whether, but when. Will the maturity of the information age come in the first decade of the 21st century or will more spectacular enterprise failures be required?
What do you think? Let me know at Larry.English@infoimpact.com or on the IQ Forum under IQ Resources at www.information-quality.com.
For more information on related topics visit the following related portals...
Larry P. English is president and principal of INFORMATION IMPACT International, Inc., Brentwood, Tennessee, and the author of the widely acclaimed book, Improving Data Warehouse and Business Information Quality: Methods for Reducing Costs and Increasing Profits. English is cofounder of the International Association for Information and Data Quality (www.iaidq.org). English is an internationally recognized speaker, teacher, consultant and author and may be reached at firstname.lastname@example.org or through his Web site at www.infoimpact.com. For more on how to improve your IQ principles and techniques, and prevent your organization from wasting millions in information scrap and rework, join the IAIDQ (visit www.iaidq.org).