Data Migration Part II : Evaluation & Planning

As Antoine de Saint-Exupéry said, “A goal without a plan is just a wish.” In terms of Data migration that is so true,  according to Gartner, 83 % of data migrations fail outright or exceed their allotted budgets. A lot of this failure is attributed to a lack of planning.

Analysis of data migration projects over the years has shown that they meet with mixed results. While mission-critical to the success of the business initiatives they are meant to facilitate and support lack of planning structure and attention to risks causes many data migration efforts fail.
— Gartner, “Risks and Challenges in Data Migrations and Conversions,” February 2009, ID Number: G00165710

In the last blog, I mentioned about Data Migration stages. In this blog will do a deeper dive and look in the first 2 stages,  So let’s talk about what planning and Evaluation look like.

Evaluate: Data

  1. Data Sources: Identify all different data sources for Data migration, often time Dynamics 365 implementation replaces more than one applications as such you might be migrating data from multiple applications. 
  2. Data Quality: it’s important to analyze the quality of data at the source to avoid Garbage In, Garbage Out. Some of the common quality issues are
    1. Inconsistent and incomplete Data: inconsistency is a big indicator that there’s a data quality problem. When data has not been entered in the system correctly like missing zip code, province or other information remaining data could be of less value.
    2. Duplicates: In many circumstances, the same records might exist multiple times in a database, Common culprits are Account, Contacts.
    3. Inconsistent Formats: if source application did not have much validation the data will have more typos of errors, Address is the most common place to find with City and provinces being named differently.
    4. Obsolete Data:  Every day, people move, marry, and change their names. Since Peoples’ information is rarely static, obsolete data is a common challenge which is hard to rectify as well.
  3. Data Volume: Volume of Data could be a key factor in determining Data migration Tool/Process and the Deployment Plan.
  4. Operational Data vs Archival Data: Legacy applications when replaced will often have a lot of historical data, you might not want to bring all of this data into Dynamics 365 as this data might be not be used for day to day operation but useful for Data Analytics and Reporting Purposes. Identify what data needs to be in Dynamics 365.

Continue reading “Data Migration Part II : Evaluation & Planning”

Dynamics 365 : Data Migration demystified-Part I

Data migration is often overlooked in Project implementation primarily because the project team is focussed on application development and customization which takes away the priority of data integration and data migration. As data migration is planned to be the last activity on the project it also gets most impacted from any budget cuts and application scope increase. 

Talking about the budget, many implementation’s approaches to data migration is for data migration to be done using the Import and Export feature of Dynamics 365 CE over the weekend before Go Live, this works sometimes but is a recipe of disaster in most cases. 

When implementing Dynamics 365 Project, configuration and customization to meet the users need is only half the battle. Data migration is key to getting the right head start for users to start using the application. In this blog series, I will focus on data migration, the challenges, strategy, Process, Tools and my own learnings over the years. Let’s start with what is Data migration?

Continue reading “Dynamics 365 : Data Migration demystified-Part I”

Unified View in Dynamics 365 without Data Duplication – Part 2

In the previous blog  – we looked into how to achieve a unified view in Dynamics 365 CE using USD and Virtual entity, continuing on let’s look into other available options, today we will focus on Dynamics 365 CE Form embed options. Continue reading “Unified View in Dynamics 365 without Data Duplication – Part 2”

XrmToolBox: Portal Webform Cloner

When building/configuring Dynamics Portals for collecting data from users we generally use entity form or a webform.

Multi-Step webform is one of the key features of Dynamics Portals. It allows for multi-step conditional forms to be created in different modes clubbed together. 

More often than not, you need forms to do things differently. Allow users to create and submit details. Allow users to edit the submitted details and present submitted details as read-only.

But this is difficult to achieve with one webform. The mode of the webform step is fixed. It can either be Insert, Edit, Read-only. Continue reading “XrmToolBox: Portal Webform Cloner”

Global Integration Bootcamp – Calgary

Global Integration Bootcamp is a global scale, integration focussed Bootcamp hosted and organized locally and backed by Microsoft. It is held across the world in over 25 cities. The event focuses on all Azure Integration services and applications.

This year I had an opportunity to bring this event to Calgary and Host and organize it along with Dhina Gajavarathan. This was my first time organizing such an event and I wasn’t sure how it will go. To start with we had two speakers Myself and Dhina, But I can’t be thankful enough to the Community who came out to support it.

We published the event on Friday afternoon and by  Monday we had 3 Microsoft MVP’s as Speakers and 25+ attendee already registered.

With Kent Weare, Shamir Charania and David Drever coming onboard as speakers for the event, we were certain that we will have more registrations coming,  We tried to move the locations to Microsoft Office but being an after office weekend event made it a challenge.

Nasir Nasiruddin from Microsoft helped us out and we were able to move the location to MTC, Microsoft Office, which is the perfect location for hosting an event like this. This was key as I got to know later that few of the cities had to cancel the Integration Bootcamp event as the team not able to secure a location for the event.

Event Day was fun, we started early to set things up. Our friend Sarvesh Thuse was of great help, helping us with setup, prep, managing Registration Booth and guiding Attendees.

We had a great Event with very engaged attendees, thanks to awesome speakers and engaging Agenda for the day. We had high participation(40 attendees out of 53 registrations) and attendee engagement in our Q&A Panel discussion.

Here are some of the pics from the Event.

[ngg src=”galleries” ids=”1″ sortorder=”19,12,25,23,4,1,2,3,5,6,7,8,9,10,11,13,14,15,16,17,18,20,21,22,24,26,27″ display=”basic_slideshow” arrows=”1″ interval=”2000″ show_thumbnail_link=”1″] Continue reading “Global Integration Bootcamp – Calgary”

Unified View in Dynamics 365 without Data Duplication

Data integration are meant to solve two primary problems.

  • Unified data by creating a co-relation between data available in different applications.
  • Unified View by allowing users to see a unified view of the business data without hopping from one application to another.

In this blog post, we will focus on Unified View. One of the shortcomings of achieving a unified view by data Integrations is data duplication. Even though the Master data is stored in one place, a copy of its subset exist in other applications in order to create a Unified view. This data needs to be synchronized and updated as the master data changes. In some scenario, the changes from downstream application might update the master data and other copies.

Often time Dynamics CE is used to maintain Customer data while Operation data and Financial data are maintained in another application. in order to access the data, users need to navigate to the relevant application and look up the data. As an example, you might have customer record and Order data in CRM but the invoice and payment data could be in another ERP system( AX, GP, SAP etc.).

In Dynamics 365 CE there are multiple ways to achieve Unified View without or with reduced duplicate data, let’s dive into what these solutions are and when to use them.

Continue reading “Unified View in Dynamics 365 without Data Duplication”

Monitoring Microsoft On-Premise Data Gateway

There are more and more installs of Microsoft On-premise data gateways to achieve Hybrid integrations scenarios. On-premise gateway provides a
quick and secure data transfer between on-premises data and Azure Cloud Services. In this blog, we will look into how we can monitor On-premise gateway Install, but before we get to it lets look into what is on-premise data gateway.

Whats is On-Premise Data gateway?

It allows you to build integrations which securely moves data between on-premise data sources and Azure Cloud Services.

Currently, on-Premise data gateway can be used to interact with Azure Service Bus, Azure Logic apps, Microsoft Flow, PowerApps, Power BI and Azure Analysis services on Azure. Here is the list of all different supported data sources.

How it works ?

Continue reading “Monitoring Microsoft On-Premise Data Gateway”

D365UG Summit 2018, Phoenix

The last couple of weeks had been really busy, I had an opportunity to attend and present at two conferences events by Dynamics communities.

Continuing on my year of firsts, I attended  D365UG Summit 2018 conference at the beautiful and vibrant city of Phoenix from Oct 15- Oct18.

User Group Summit is one of the biggest Dynamics conferences, The conference is aimed at bringing together all the Dynamics 365 users, Customers, Partners together and sharing the knowledge and learnings. The conference is a confluence of different Dynamics conference(AXUG, GPUG, BCUG/NAVUG) geared towards different Microsoft Dynamics products in the Microsoft Dynamics ecosystem, this enables attendees to not only learn more about the Dynamics product they have been working with but also learn and explore other Dynamics capabilities.

Continue reading “D365UG Summit 2018, Phoenix”

My First Ignite-2018

This has been the year of a lot of First for me.  Last week  I attended my first Microsoft Ignite conference at Orlando, Microsoft Ignite is one of the biggest Microsoft Tech conference of the year. It was attended by 30000+ technology enthusiast from across the world.

Microsoft Ignite and Microsoft Envision were both hosted at the same time at Orlando. while Ignite is more focused towards IT Professional s and Developers, Envision is for Business leaders. KPMG was a platinum sponsor of Envision.

Ignite is the best place to learn, Network and meet Microsoft Engineers, Partners, vendors, and technology enthusiast while having fun, As the conference focuses on all the Technological offering from Microsoft, it allows you to learn about the different areas from office apps to Business Apps, Azure,  AI, and IOT etc..

Continue reading “My First Ignite-2018”

Migrate Adxstudio portals to Dynamics 365 Portals

It’s been a more than a month since legacy Adxstudio portals support officially ended.  Many customers and partners are still using Adxstudio Portals and yet to migrate to Dynamics 365 Portals or its short-term alternatives( Microsoft Open Source Portals, xRM Portals Community Edition).

As there are multiple Portals available, there is always a question on which one to migrate to, we currently have 4 different versions of Portals:

    • Dynamics 365 Portal Capabilities is the only viable long-term option, it’s Dynamics 365 Online dependent and only available as Software as a Service solution. This is the version you want to be using.
    • Microsoft Open Source Portals is the 8.3 version of Dynamics 365 Portals, it’s completely open sourced and released under MIT license. This is a one-time release and no future bug fixes, updates or patch will be provided. It is suited as a stepping version before upgrading to CRM Portals. I wouldn’t recommend being on this version unless you and your team has an in-depth understanding of the Portals and can self-support.
    • xRM Portals Community Edition is the githubbed version of Microsoft Open Source Portals, it is hosted by KPMG Adoxio and supported by Adoxio and Portals community, This is the version for customers or Partners who need sometime before moving to Dynamics 365 online or are waiting for new releases/features on Dynamics 365 Portals. This is not an on-premises option for Dynamics Portals but a short-term interim self-hosted option.
    • Adxstudio Portals, Though the support for Adxstudio Portals has ended, if you happen to have a perpetual license you may continue to use the product knowing it’s not supported.

The only long-term option is to upgrade to Dynamics 365 Portals. Dynamics 365 Portals and AdxStudio Portals don’t have same Portal website offering, some of the key differences are:

Logic App Definition Parameter Vs ARM Parameters – ALM Part 3

In the previous blog – we looked into getting Logic Apps CI/CD using ARM Template Parameter, we briefly touched on logic app definition parameter. In this post, we will look further into using Logic app definition parameter as an intermediary and the benefits of doing so.

Comparison:

ARM Parameters Logic App definition Parameters
Used with ARM Template Used within a Logic App
Value evaluated during Deployment Value evaluated during Runtime
Syntax: “uri” : “[parameters(‘vsts_bug_url’)” Syntax:  “uri”: “@parameters(‘url’)”
Not referenced post-deployment Available in Code view and Designer view
Suited for Environment Specific values Suited for Reference objects

so the obvious question is how we can leverage Logic App parameters as an intermediary and why to do so?

Continue reading “Logic App Definition Parameter Vs ARM Parameters – ALM Part 3”

Logic App ALM Process for Dynamics 365 Integration – ALM Part 2

In the previous blog  – we looked into working with Logic Apps from Visual Studio, which allowed us to manage logic app code in TFS or Git or any other source control system. But to truly do DevOps for Logic Apps while integrating with Dynamics 365 you need to be able to do ALM process for Logic Apps better yet do continuous delivery/deployment.

when we are editing and deploying a Logic app in Visual Studio we are actually editing an ARM Template, So what is an ARM template?

ARM templates are JSON format template, it enables you to deploy Azure resources in a declarative manner and could be used to define all the resources in a resource group.

An Empty ARM Template Schema

{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": { },
"variables" : { },
"resources": [],
"outputs": { }
}

To add a resource in ARM template, resource details are declared under resources properties, there could be multiple resources. Below is a sample of Empty Logic app added to empty ARM Template.

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {}, //ARM parameters
  "variables": {}, //ARM template variables
  "resources": [
    {
      "name": "Logic APP Name",
      "type": "Microsoft.Logic/workflows",
      "location": "West US",
      "tags": {"displayName": "LogicApp" },
      "apiVersion": "2016-06-01",
      "properties": {
        "definition": {
          "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
          "contentVersion": "1.0.0.0",
          "actions": {},
          "outputs": {},
          "parameters": {},//Logic App defination parameters defination
          "triggers": {}
        },
        "parameters": {}   //parameter values
      }
    }
  ],
  "outputs": {}
}

Continue reading “Logic App ALM Process for Dynamics 365 Integration – ALM Part 2”

Working with Logic Apps from Visual Studio

Azure portal makes it really easy to get started with and creating and managing Logic Apps, with all the built-in connectors and easy interactive orchestration UI you can be up and running with your integration in a couple of hours.

But what if you want to manage your code in TFS or Git or any other source control system. Working and creating Logic App in Azure is great but as DEv we enjoy the comfort that Visual Studio offers.

Good news is that you can create manage and deploy logic App from within Visual Studio and it’s better if you have become familiar with creating Logic Apps from Code View Interface.

In order to get Started with Logic App, you need to install Azure Logic apps tools for VS.

Once Installed you are ready to write your first Logic App from VS, Start by Creating new Project of type Arm.

Once you select your Project you will be asked to select the Type of Azure component you are creating. For our purpose, we will select Azure Logic App.

Continue reading “Working with Logic Apps from Visual Studio”

Migrate Dynamics 365 Notes Attachments to SharePoint

Quite often we run across a Dynamics 365 implementation where not much thought was giving to Document Management while implementation, resulting in user using Notes(Annotation) entity for document management.

Notes do offer a lightweight solution for managing and uploading documents to related record but it lacks the capabilities of a full-fledged document management capabilities. Notes are also stored in CRM database, storing documents as notes could easily become a costly endeavor considering Dynamics 365 storage cost. Jukka Niiranen has written a great blog on file storage and disadvantages of using Notes/Annotation.

The first step to transition from using CRM Notes to SharePoint for document management is migrating existing documents from Notes to SharePoint. This may seem like a daunting task but could easily be accomplished using SSIS package for migration.

I have an SSIS starter project on GitHub which illustrates the migration for notes related to Case entity, It could be easily configured to migrate notes related to any other entity.

SSIS by itself doesn’t provide capabilities to connect with Dynamics 365, for this, we use a third-party connector like KingswaySoft or CozyRoc, in below example we are using KingswaySoft connector, license and pricing details could be found here. Kingsway does provide the developer license free of cost.

Continue reading “Migrate Dynamics 365 Notes Attachments to SharePoint”

eXtreme365 Dubrovnik 2018

Extreme365, just completed in Dubrovnik, Croatia. What an amazing, beautiful and fun location for a great Dynamics conference. A week surrounded by amazing people sharing knowledge while having fun.

This was special for me as this was my first Extreme 365 conference speaking engagement. I presented my take on “Modern Approach to integration with Dynamics 365“, the session was well attended and very well received. Thank you all who attended my session.

The conference coincided with spring release announcement of Next version/ generation of Dynamics 365. This was one of the biggest Dynamics release announcement. for the first time, Microsoft had to release document rather than release notes primarily because of the number of changes being shipped on this release. With this release Dynamics, 365 becomes the Application platform for business application.

We also watched live streaming of  Business Forward’s in Amsterdam with Jean-Philippe Courtois, Alysa Taylor, and James Phillips at the conference.

Key Highlights from Dubrovnik: Continue reading “eXtreme365 Dubrovnik 2018”

Migrating Microsoft Flow To Azure Logic App Service

With the increasing popularity of Microsoft Flow and ease of setup quite often we start with a self-service integration using Microsoft Flow and later need to switch them to Azure Logic apps.

As Flow is built on the top of Logic Apps it’s a simple export-import(Export and Deploy) process to switch them over. In this blog, we will go over converting an existing flow integration to Logic Apps Service.

I have built a Flow which is triggered when a case is created in Dynamics 365 and in turns creates a work item in Visual Studio Team services for the team to work on. As a lot of Cases are being submitted I will switch them over to Logic Apps for better monitoring.

In order to migrate my first step is to export this Flow as Logic Apps from My Flow Page. Continue reading “Migrating Microsoft Flow To Azure Logic App Service”

Dynamics 365 Portals: Create new entity record from within Lookups

If you have used Dynamics 365 you must be familiar with the + New button on the lookup. This enables the user to create new item instead of selecting exiting one.

This feature has been a frequent ask for Dynamics 365 Portals. Good news is that this functionality was added by the product team few releases back the bad is that it is only partially done.

In this post, I’ll walk through the steps for configuring it and its limitation.
Continue reading “Dynamics 365 Portals: Create new entity record from within Lookups”

Microsoft Dynamics 365 Connector for Logic App gets an update

Last week Microsoft released an update for Dynamics 365 Connector for Logic App, there are few long-awaited features in this update which I am excited about.

With this update Microsoft has added support for option sets being exposed as strings in the response, this used to be a challenge earlier as Optionset value and mapping had to be hardcoded in the Logic app or Azure Functions.

the difference in the options set value in the output:


Continue reading “Microsoft Dynamics 365 Connector for Logic App gets an update”

Integration with Microsoft Dynamics 365 using Azure Logic App Service

In this post I’ll discuss Data Integration using Azure Logic App Service, we will create a Logic app that gets triggered when a case is created in Dynamics 365 and inturns creates a work item in Visual Studio Team services for the team to work on.

Logic App workflow is built using series of actions and Connectors, and has its own JSON based workflow definition schema, for details refer to my blog on The Basics of Azure Logic Apps Schema and Codeview editing

Pre-requisites
  • Microsoft Dynamics 365 Online instance (On-premise instance could be used as well using Azure Data Gateway)
  • Microsoft Azure subscription for creating a Logic APP (If you don’t have an Azure subscription you can either use Try a Logic App feature which creates a logic app for an hour or you may sign up for a Trial account.)
  • Visual Studio Team services account (SignUp link).

Continue reading “Integration with Microsoft Dynamics 365 using Azure Logic App Service”

Basics of Azure Logic Apps Schema and Codeview editing

Integrations using Logic App could be a fun exercise. Logic App provides a visual designer to model and automate your workflow process as a series of steps called as workflow designer UI,  for complex requirement Code view comes handy. In this blog, I will talk about the structure of logic app definition.

Let’s start with some basic components of Logic App.

    • Logic App Visual designer actually creates a JSON file with workflow definitions. JSON structure for Logic app

{
"$schema": "",
"contentVersion": "",
"parameters": { },
"triggers": [ { } ],
"actions": [ { } ],
"outputs": { }
}

$Schema, content version and static values and are set to schema and version automatically.

Parameters are important as they help parameterize the logic app and assist with Deployments and change management. I will go more in-depth in my subsequent blog on Logic App and Change management/ALM.

Triggers: define the trigger criteria for the workflow. to access the output of trigger @triggerBody() could be used.

Actions: define the actions within the workflow, actions are nested and using runAfter you may define the dependent action or parallel action flow.

Outputs: refer to the response of the workflow post execution of all the actions. Most of our Integration scenario doesn’t use the output as we will have another action to consume the output.

Continue reading “Basics of Azure Logic Apps Schema and Codeview editing”