Arbutus Blog

Agile Technologies and Data Warehousing

“An ounce of planning is worth a pound of prevention”. In the DW world, this pearl of wisdom could not be more true. Many traditional data warehouse projects require months of planning before the earliest implementation steps are achieved.  Users do not get a chance to view or vet the implementation until many resources and much time have already been spent. Although there can be many reasons for this, it can be argued that one of the key contributing factors for this inefficiency is the implementation methodology itself, especially at the start of a project.

An idea growing in acceptance is the adoption of agile technologies at the earliest stages of implementing a data warehouse or data mart. This has proven to reduce the timelines for data warehouse or data mart projects. One of the key advantages with agile technologies is that it enables you to profile, test and see your data warehouse or data mart before it’s actually built.

Using an agile technology, before the first implementation steps, users are provided with immediate access to core information.  Users can immediately see the results and provide instant feedback.  This feedback can be easily incorporated, yielding an instant warehouse model that reflects the evolving scope and requirements of your data warehouse project.  If needed, the agile technologies allow you to choose to continue using the prototype for a period of time, to ensure that the requirements have stabilized before you undertake the full implementation.

Requirements of agile technologies for data warehousing:

• Directly reading real source data, including complex mainframe legacy sources when needed
• Providing real interfaces to end-user applications
• Integrating any number of disparate data sources into a single logical view

Key features of agile technologies for data warehousing:

• The ability to fully implement business rules
• The ability to perform any data mapping or profiling
• The capabilities to implement any necessary data cleansing or conversion requirements
• The flexibility to add new or modified tables, columns, or data relationships
• The adaptability to modify or add cleansing, transformation or mapping rules
• The ability to provide filtering
• The ability to perform dynamic calculations

Important benefits of agile technologies for data warehousing:

• Saving time to model and implement the data warehouse
• Saving money by minimizing the subsequent re-work required
• Ensuring end-user buy in by incorporating end-user feedback early and often to create a functional and acceptable model

Once the final model has been established and has achieved acceptance, implementation can proceed using standard data warehousing tools and techniques. As the project proceeds, there can be a much higher comfort level with the knowledge that the users have already seen, worked with and participated in the development of the final data model.

Agile Data Migration, is this possible?

The harsh statistics are that 67% of all data migration projects are not delivered on time and average a 41% time overrun.

Eleanor Roosevelt said “Learn from the mistakes of others. You can’t live long enough to make them all yourself.”

What we have learned is that the traditional/common approach to data migration projects relies upon developing conversion programs in 3rd GL languages or involved ETL tools that typically take many weeks or months to research and code.  Often data quality issues are not found until late in the process when the conversion programs are finally being run and tested. This often results in further delays as missing or additional data conversion requirements are uncovered late that need to be programmed into the conversion process.

As a result, many companies struggle with the complexities of migrating legacy data into new systems like SAP or creating a new data mart.  There are a variety of reasons for this including:

• A lack of legacy system skill sets and tools to understand and work with the source data
• An inability to foresee and validate all the data quality issues and business rules within the source data so that data problems are often not identified until the data migration is already well underway
• Rigidity in the overall process that prevents, delays, or makes very costly implementing changes to address unexpected data issues or changing requirements.

Increasingly, companies faced with data migration projects are looking into using more agile technologies to enable a much more proactive approach to data migration and to allow emerging issues or changing needs to be dealt with immediately.

For Agile technologies to be effective in a data migration or data warehousing project they need to:

• Provide access to source data directly from the legacy systems
• Enable data querying/profiling to proactively discover data quality issues and to validate/determine business rules in the source data
• Perform the necessary data transformations required for loading data into the new system
• Facilitate timely changes to the migration process based on emerging issues or changing needs

Is Agile Data Migration possible? In a future post will be a real-life story from last year that would suggest, yes, Agile Data Migration is possible.