Data migration is often overlooked in Project implementation primarily because the project team is focussed on application development and customization which takes away the priority of data integration and data migration. As data migration is planned to be the last activity on the project it also gets most impacted from any budget cuts and application scope increase.
Talking about the budget, many implementation’s approaches to data migration is for data migration to be done using the Import and Export feature of Dynamics 365 CE over the weekend before Go Live, this works sometimes but is a recipe of disaster in most cases.
When implementing Dynamics 365 Project, configuration and customization to meet the users need is only half the battle. Data migration is key to getting the right head start for users to start using the application. In this blog series, I will focus on data migration, the challenges, strategy, Process, Tools and my own learnings over the years. Let’s start with what is Data migration?
Data Migration: is the process of transforming and moving Data from Legacy Systems to the Target System.
In Dynamics 365 we move data for two different purpose :
Configuration migration: is moving configuration data from one implementation environment to other (Dev -Test- UAT-Prod), This migration often consist of moving data in application-specific configuration entity( Settings, Queues, Themes, Portals content, VOC setting, etc.). Configuration data migration should be built in into ALM/DevOps process in the project, Configuration migration tool in the D365 SDK is well suited for this.
Legacy data migration: is migrating data from the legacy or other applications to Dynamics 365. In some cases, the legacy application might be on D365 platform where you may be reimplementing or moving from on-prem to online. This requires heavy data lifting while ensuring data integrity is maintained, This is best done using an automated script/package to perform the ETL process.
In this blog series, I will focus on Data migration, the challenges, strategy, Process, Tools and my own learnings over the years. Successful data migration is largely dependent on the amount of planning and strategy that goes into it. Migrating data to Dynamics 365 is not easy, Data migration in general has a lot of challenges but working with Dynamics 365 brings its own challenges to keep into consideration.
Challenges:
-
- Can’t write to SQL directly, have to work with SDK: when migrating to D365 all data operations happens through SDK and SDK helps us by abstracting the complexity of underlying data structure D365 CE uses but brings other challenges.
- Lookups use GUID: All the associations happen through GUID. often generated by CRM.
- Lookups use multipart fields( Owner: OwnerId, Ownertype)
- Relationship dependencies, circular reference: D365 often has a lot of circular references where the same type of records are linked with each other using Parent-Child relationship.
- Data integrity: carrying over metadata fields, created by, created on, modified by and modified on.
- Special Entities: D365 has special entities which require different handline like incident resolution, order close, etc.
- Migration Error: Plan for error handling, failed rows and reruns
- Performance: data migration could take a while to complete, as such performance becomes the key.
Strategy & Process:
The data migration process can be broken down into 6 major steps/stages, each of these stages is focussed on a specific activity and are equally important.
Evaluate: this is the first step of the process where we evaluate the intent of data migration and define a Scope for the data we are trying to migrate.
Plan & Analyze: defining the strategy for data migration is the key to the migration to be successful.
Extract: define and perform the Data Extraction from different data sources, You may or may not use a Staging Database.
Clean: data cleanup is a continuous process and while it should be happening in parallel with all other steps, This stage in the last opportunity to clean the data before loading it in D365.
Load: In this stage, we execute the process to move the data into Dynamics 365, This requires a lot of planning and prep to minimize the impact on the users and maintain business continuity.
Verify: like cleanup, data validation is a continuous process as well, validation is required when data is being extracted to ensure all the requisite data is accounted for, validation is required during cleanup to ensure relevant data is not being removed and at the end, validation is required to ensure data has been loaded correctly.
This is the first blog in the series for Dynamics 365 data migration. In the subsequent blogs, I will be going over each of these stages in detail.
Let me know how you have managed data migration in your project, I would love to hear your thoughts.