Arbutus Blog

Agile Data Migration, is this possible?

The harsh statistics are that 67% of all data migration projects are not delivered on time and average a 41% time overrun.

Eleanor Roosevelt said “Learn from the mistakes of others. You can’t live long enough to make them all yourself.”

What we have learned is that the traditional/common approach to data migration projects relies upon developing conversion programs in 3rd GL languages or involved ETL tools that typically take many weeks or months to research and code.  Often data quality issues are not found until late in the process when the conversion programs are finally being run and tested. This often results in further delays as missing or additional data conversion requirements are uncovered late that need to be programmed into the conversion process.

As a result, many companies struggle with the complexities of migrating legacy data into new systems like SAP or creating a new data mart.  There are a variety of reasons for this including:

• A lack of legacy system skill sets and tools to understand and work with the source data
• An inability to foresee and validate all the data quality issues and business rules within the source data so that data problems are often not identified until the data migration is already well underway
• Rigidity in the overall process that prevents, delays, or makes very costly implementing changes to address unexpected data issues or changing requirements.

Increasingly, companies faced with data migration projects are looking into using more agile technologies to enable a much more proactive approach to data migration and to allow emerging issues or changing needs to be dealt with immediately.

For Agile technologies to be effective in a data migration or data warehousing project they need to:

• Provide access to source data directly from the legacy systems
• Enable data querying/profiling to proactively discover data quality issues and to validate/determine business rules in the source data
• Perform the necessary data transformations required for loading data into the new system
• Facilitate timely changes to the migration process based on emerging issues or changing needs

Is Agile Data Migration possible? In a future post will be a real-life story from last year that would suggest, yes, Agile Data Migration is possible.