FREE DM Review Site Registration!
Sign-up today and access DM Review on the Web!

Your FREE registration entitles you to:
FREE email newsletters

FREE access to all DM Review content

FREE access to web seminars, resource portals, our white paper library and more!

   

Publisher reserves the right to serve qualified requesters only.

Why Your Company's Data Migrations are Really Your Business

Your company is launching an important new business initiative and is implementing a cutting-edge new system to support it, and now they are preparing to migrate your data - the vital information you, your team and your peers need to help run the business. Be afraid. Be very afraid!

There are any number of reasons for migrating critical business data: a new customer relationship management (CRM) process or enterprise resource planning (ERP) system, a merger or acquisition, a new regulatory twist, the ramping up of a new outsourcer or software-as-a-service provider, to name a few. There is also any number of reasons for a data migration to go dreadfully wrong, which they do with alarming frequency. Data migrations are inherently difficult and risky undertakings. Analysts estimate that almost 80 percent of data migration projects fail to come in on time or on budget, while 33 percent fail totally. And when a migration project falters or fails, the rippling effect can cause the overarching business initiative to fail as well, leaving the business stakeholders (i.e., you and everyone else who relies on the data) in the lurch.

What most successful migrations have in common is business involvement - i.e., they are business and not technology led. Successful projects recognize that data migration is not a purely technology issue. IT will invariably (and rightly) ask, "How do we migrate all this data?" But there is a more fundamental question that has to be asked first, and that is: "What data should we be migrating?" Only the business can answer this essential question. Without this guidance from the business, IT does not have the necessary knowledge to answer the latter question, and it is patently unfair and unwise to place that particular onus upon them. The business stakeholders must take responsibility for a successful migration, or else be prepared to live with the consequences.

Case Study One - A Cautionary Tale

Tolstoy famously wrote that all happy families are alike, but every unhappy family is unhappy in its own way. Much the same can be said of data migrations. There are many ways that a migration can go wrong with many adverse consequences, but there is really only one measure of success: that the data "works" in the new system, and that business users trust its quality and completeness. That said, how can a data migration veer off track?

To help answer this question, envision yourself sponsoring the implementation of a new CRM system. There is a significant business drive to promote customer awareness, and from the CEO down this is seen as this year's number-one strategic objective. Succeed and your career will soar. Fail... well, failure is not an option.

Although you have never led this kind of charge before, you have gathered around you a collection of experienced project veterans and are confident in your ability to manage. Your external system integrator (SI) is among the best and the software that has been chosen is in the upper quartile of every industry analysis. How can you fail?

The contracts have been drawn up and the SI has shown you slides of how the custom-built migration program that will migrate the data from the old to new system will work. It's not written yet, but there is time allowed in the plan for writing and testing, and your local IT department seems to approve, so you establish your program committee, set your reporting lines, and get started.

The nine-month project is now in its sixth month. There have been a few delays getting specifications signed off, and the sales team does not seem to want to play ball, but then, this system will provide executive insights that were previously exclusively their domain. You bulldozer your way through their objections with the whip hand of superior executive sponsorship. Then the first inkling of the disaster that is about to overtake you starts to emerge.

The project leader responsible for data quality buttonholes you in the corridor. It seems that some of the data fields do not have the values in them that they need. You take away the issue that there is something missing from the accounts data. When you raise this later with the SI program manager, he promises to look into it and get back to you.

It all goes quiet for a few days. Then, at the next program meeting, there is a clear tension in the air. The SI manager responsible for data migration has added an issue to the issue register: there are data gaps in the live data being delivered. Apparently, some orders are without the correct tax code, and these orders cannot be linked to a salesperson because the original sales code has a value in it that the new software cannot recognize. But what about that custom-built migration program, you ask. Well, the SI folk reply, they are responsible for the agreed data mapping specification to the new system, but have no responsibility for the data they are getting.

So you turn to the IT department representatives, who look stony faced and resentful. This is the data as it was exported from multiple sources, principally, the legacy sales order and accounting packages. These semantic challenges are not ones they can resolve. They do not know what the correct values should be.

Wait, it Gets Worse

"Semantic challenges" is a phrase you are going to be hearing a lot over the next few weeks. It seems that as soon as one emerges, a dozen more follow, each one as perplexing as the one before. How could these systems have been running this company? You can almost hear the drawbridges being drawn up around you. The SI is retreating behind its contract - providing data of the right quality is the responsibility of your company. The IT department is making disparaging comments about the "users' data." Your old friends in sales are being horribly friendly but ultimately unhelpful: they provided all the resources asked and have no budget for additional staff to resolve the issues you are raising. Their systems work okay. Are you not sure it isn't your technology?

As the clock ticks, compromises are suggested. Surely an 80 percent success rate with your existing customer base is acceptable? The rest can be phased in as you cleanse them. But there is no money in the budget for cleansing that goes into the next fiscal year. Anyway, the company's new customer-centric business model has been unveiled to the financial analysts and this is a key plank. Delay is unthinkable.

You have no option but to command the IT department to fix what they can. It is possible, they suggest, to "default" some of the missing values. "Default" you learn is another word for "guess at." The results are not promising, and the little that the end user departments have seen of the new system is engendering unfavorable comments.

You could "postpone functionality," it is suggested. The customer profiling software and the Web front end are "obvious candidates" for later implementation. But the whole business case was dependent on customer profiling being available and the lack of a Web front end would be an obvious public failure.

At the end of day, you have no choice but to take these unpalatable truths to your management. The response is as you anticipated. You are instructed to carry on, but you hear the words "trouble shooter" and you know your time is limited, along with your career.

How did it all go so wrong? At the outset, the software people, both internal and external, seemed confident in their prowess and products. So where did these data gaps and semantic issues come from? And how did they emerge so rapidly yet so late in the project that recovery was impossible? Most importantly, what could have been done differently and better?

Threading the Migration Maze

As this scenario illustrates, data migrations are fraught with peril. Typically, right off the bat, organizations face the challenge of insufficient data migration expertise coupled with insufficient business ownership. Once the project is underway, they come face to face with a general lack of understanding of source data and systems - after all, the people who created them are usually long gone, and the documentation is seldom complete and up-to-date. Then comes the challenge of imperfect data validation - legacy data is often plagued by gaps, redundancies, inconsistencies, damage caused by long-forgotten bugs and their fixes, and so on, and it must be remedied and validated. Finally, the challenge of supporting the entire migration lifecycle comes - a data migration doesn't end when the new system goes live, rather, its legacy lives on in the ongoing effectiveness, or lack thereof, of that new system. What you do upfront, for good or ill, will resonate for a long, long time.

Any one of these challenges by itself can rock a migration project. Often they combine in a perfect storm to sink it. In the best case, users are alienated and timeframes and budgets are blown. In the worst case, the new system doesn't deliver as promised, users can't trust their data and reject the new system, and the projected benefits of the overarching merger or acquisition, new line of business, state-of-the-art enterprise reporting infrastructure, etc., are severely and perhaps irrevocably compromised.

Yet there is a way through this perilous maze: "four golden rules" of data migration.

Rule One: Data migration is a business not a technical issue.

A data migration is always designed to meet a business goal, and only the business is able to know how completely that goal has been met. In confusing technical know-how with business needs, there is always the risk that the focus of the project will drift away from the business goal toward what the technologists are best suited for and pleased to deliver. This occurs even with the best intentions in the world because the technologists cannot fully understand the enterprise significance of the data or the business drivers. The business is the ultimate arbiter of success .

Rule Two: The business knows best.

Say it as if it is your mantra. Business stakeholders cannot abdicate responsibly for migration decisions. To do so is to leave useful, hard-won knowledge on the table and bow to simple expediency. There are data owners and users in the enterprise who, through hands-on, day-to-day experience, know where all the necessary data is, realize its issues, and understand what needs to be migrated. The business thus has the familiarity and expertise to make the necessary informed judgments regarding data quality and appropriateness.

Find these people, engage them and make certain they are listened to. They will be vital to helping validate and ensure the quality and completeness of the data that ends up in the new system - not just at "go live," but across the lifetime of that system.

Rule Three: No one needs, wants or will pay for perfect data.

This may seem counterintuitive given the importance of high quality data. Yet many migrations flounder because of an excessive emphasis on achieving data perfection. Given the rigorous deadlines of a migration project, you can easily run out of time and end up compromising data quality and jeopardizing the project in a mad dash to the finish line, unless you are willing to put a stick in the ground at the start and say, "This is an appropriate level of quality; let's shoot for that."

The data owners and users know the level of quality that is appropriate and need to be engaged in inserting that stick solidly into the ground. Leaving the task to someone who doesn't know the data and its uses (i.e., a technologist) is a blatant dodge of responsibility. Similarly, leaving the task to the end of the project is an invitation to panic-based decisions that end up pleasing no one and injuring many.

Rule Four: If you can't count it, it doesn't count.

You know the quality of your legacy data is "x" and you want it to be "y" or even a lofty "z" in the new system. Fine, but how do you measure it, now and going forward? Data quality isn't static. It erodes and, conversely, it can be improved over time. But how do you keep it from eroding and how do you improve it on an ongoing basis?

In following Golden Rules One and Two, you can create measures that make sense to the enterprise, not just to the technologists. Then you can more meaningfully measure the deliverables, perform gap analyses to chart progress, and monitor and improve quality going forward. In fact, one of the key benefits of closely involving business stakeholders in a data migration is the education they will receive in driving and measuring data quality improvements long after the new system has gone live and the technologists have gone home.

Case Study Two - A Migration Done Right

Here is an example of a data migration that proceeded according to the four golden rules, a large European telecommunication company seeking to comply with a disruptive new regulatory mandate. By a given date, the company's network had to be open for competition, without fail. This meant that complex network services had to change hands, tightly linked systems had to be decoupled, and an entirely new operating model had to be up and running on time. There was literally no room for a fall-back plan, and everything had to cut over all at once.

The first two golden rules revolve around putting the business in the driver's seat and involving the business stakeholders. The telco took this to heart, and, drawing on the experience of the stakeholders, the migration team developed a "fall forward" plan so that even if the systems failed to work, the company would still be compliant using a largely manual solution. With that aspect covered, the actual migration work began.

Not surprisingly, not all of the required data was in the required shape. The systems that had to be decoupled were originally designed to work together, not alone. Gaps had to be filled, etc. Again, the stakeholders stepped in to help prioritize these activities and set the desired data quality levels.

As it turned out, the quality levels determined by the stakeholders were exacting. But they were not open-ended so as to invite an undue emphasis on achieving absolute perfection. To ensure there were no show-stopping data flaws that would mysteriously arise at the last minute, the team meticulously analyzed and measured the source data, and cleansed, validated and transformed it until it all reached the business's pre-determined, ready-to-migrate level.

At the eleventh hour of the aggressive project schedule, all the data loaded perfectly, the systems cut over seamlessly, and everything worked as planned. Compliance was achieved, and the migration was deemed a success - and a testament to the combination of business ownership and knowledge-driven IT work.

Your Data is Your Business

At the beginning of this article, we identified business stakeholders as being everyone with a stake in the data being migrated - i.e., all those with a legitimate interest in the migration outcomes, including yourself. This key constituency needs to be involved from the start. They include, in descending order of criticality: data store owners, who have the power to decommission legacy systems in the wake of a successful migration; business domain experts, who understand how your organization's data sources are used; and technical data experts, who know how the legacy systems work. You also want to involve select data users as well as regulatory and audit experts.

The important thing is that they all buy into the answer to that most basic of questions: "What data should we be migrating?" When this question is answered, you can enlist these people and others to identify qualitative data issues in the legacy sources and decide how to attend to them so they do not migrate along with your data to the new system. Then and only then should the actual mechanics and technology aspects of the migration be addressed.

It is also valuable to engage in a simple, one-to-two week exercise called a migration readiness assessment (MRA). This can be kicked off just as soon as you know that a migration is pending. An MRA gives you the lay of the land before you pull out all the stops of a full-blown migration. Essentially, the business stakeholders identify one or two of the most critical data elements that need to be migrated. They highlight the issues and determine the quality parameters, and the technologists then go through the paces of mapping and validating the selected data to the target system. Like moving one or two large boxes into a new house before your real move, you discover where the tight stairways and the narrow doors are so that the rest of the move can go more smoothly. You get a head start on the migration because you possess mappings for the selected critical data sets, and you get to see how your stakeholders cooperate with the technologists, prior to the "the big show."

However you look at it - no matter what systems are involved, which technology tools are used, or what industry you are in - "the big show" belongs to the business. Keep that in mind and migration fear and loathing can give way to feelings of empowerment to help make your critical business data all that it should to be, and your vital new business initiative an unqualified success.


Arvind Parthasarathi is director of solutions at Informatica, where he is responsible for managing the company's solutions for data migration and master data management. He has wide experience with migration projects at some of the world's leading ERP and supply chain implementations. Prior to joining Informatica, he was director of product management at i2 Technologies and managed a number of solutions, including RFID, master data management, service oriented architecture, and supply chain event management. Parthasarathi is a graduate of the Massachusetts Institute of Technology and the Indian Institute of Technology. He can be reached at: aparthasarathi@informatica.com.

For more information on related topics, visit the following channels:



Industry Vendors