Migrating On prem CRM, SFA to Azure

Azure deployment, CRM and a SaaS front end application for end users. Migration of data to Azure and the Azure Data Lake.

Necessary Data Migrations

1-On Premises systems to Dynamics/CRM and SaaS front end application

• Historical, Operational data

2-HubSpot to Dynamics: Lead management data stored in HubSpot will be migrated.

Migration to Azure and Azure Data Lake

Data Extraction

• Extract data from the source systems in a suitable format. This could involve:
  • Database queries
  • API calls
  • Data exports
• For documents, extract them from the current storage solution (e.g., Pandora).

Data Cleansing and Transformation

1. Cleanse and transform the extracted data to ensure it meets the requirements of the target systems (Dynamics, SaaS, Azure Data Lake).
2. This may involve:
   • Data standardization
   • Data validation
   • Data mapping (source to target fields)
   • Data transformation (e.g., format conversions, calculations)
3. Azure Data Factory or Azure Data Lake Analytics for scalable data transformation.

Data Loading

1. Load the transformed data into the target systems:
   • Dynamics: Use Dynamics APIs or data import tools to load member data, lead management data, and other relevant information.
   • INSTANDA: Use the SaaS APIs or data import mechanisms to load membership product data, rating data, and other relevant information.
   • Azure Data Lake:
     • Store the data in the appropriate format (e.g., Parquet, Delta) within the Azure Data Lake.
     • Organize the data into a logical folder structure for efficient access and querying.
     • Azure Data Factory to orchestrate the data loading process.

Document Migration

1. Migrate the extracted documents to the new document storage solution (Azure Blob Storage).
2. This may involve:
   • Uploading documents to Azure Blob Storage
   • Metadata tagging for efficient retrieval
   • Integrating document storage with Dynamics and the SaaS

Historical Data Migration

1. The documents mention that the level of granularity migrated for older business will be reduced.
2. This implies a need to define a strategy for handling historical data, which may involve:
   • Data aggregation or summarization
   • Data archiving
   • Defining retention policies

Key Considerations for Azure Migration

Rollback strategy

Azure Services: Leverage Azure services for efficient and scalable data migration:

Azure Data Factory: For data orchestration, transformation, and loading.

Azure Blob Storage: For storing documents and unstructured data.

Azure Data Lake Storage: For storing large volumes of data in various formats.

Azure SQL Database: For storing structured data.

Data Security: Implement appropriate security measures to protect data during migration:

Encryption in transit and at rest

Access control and authorization

Data masking or anonymization (if necessary)

Migration Approach: Choose an appropriate migration approach:

Big bang: Migrate all data at once (higher risk, shorter downtime).

Phased migration: Migrate data in stages (lower risk, longer duration).

Testing and Validation: Thoroughly test and validate the migrated data to ensure accuracy and completeness:

Data reconciliation

Data quality checks

User acceptance testing

Cutover Planning: Plan the cutover to the new systems carefully to minimize disruption to business operations:

Downtime planning