When migrating data to a new platform, for example from on premises to cloud, you should fix-forward, meaning, that any data record changes are of current, not historical records.
Core Principle: Focus on the Current Record
In most operational scenarios prioritizing changes to the current record is the most practical and risk-mitigating approach.
- Operational Impact: Most business decisions and actions are based on the current state of a member’s information or policy. Changing historical records may not have any bearing on those immediate actions.
- System Performance: Updating large volumes of historical data can be resource-intensive and slow down system performance, especially in a transactional system like Dynamics 365 or similar.
- Data Consistency Risks: Modifying historical data can introduce inconsistencies if not done carefully. For example, if you change how a field was calculated historically, it might conflict with past reports or audits.
- Audit Trail Complexity: Altering past records complicates the audit trail. It becomes harder to track what the “official” record was at a specific point in time.
- Development Effort: Building the logic to selectively update historical records adds significant complexity to development and testing.
When Historical Changes Might Be Considered
However, there are specific situations where limited historical updates might be necessary:
- Data Correction (Critical Errors): If there’s a significant error in historical data that affects legal compliance, financial reporting, or member safety, correcting those specific errors might be warranted. This should be handled with extreme caution and rigorous auditing.
- Regulatory Requirements: Some regulations might mandate the correction or updating of historical data in specific circumstances.
- Specific Reporting Needs: If there’s a very specific reporting requirement that necessitates some historical data adjustment, it might be considered, but the business need must strongly outweigh the risks and costs.
Recommendations for most Programmes
- Data Governance Policies:
- Establish clear data governance policies that define when and how historical data can be changed.
- These policies should emphasize the principle of only changing historical data when absolutely necessary.
- Impact Assessment:
- For any proposed change to historical data, conduct a thorough impact assessment that considers:
- The business need
- The risk to data consistency
- The performance implications
- The audit trail implications
- The development effort
- For any proposed change to historical data, conduct a thorough impact assessment that considers:
- Auditing and Logging:
- Implement robust auditing and logging for any historical data changes.
- Record who made the change, when, and why.
- Data Archiving Strategy:
- Consider a data archiving strategy to move older data to a separate system. This can help to improve the performance of the main systems and reduce the need to change historical data.
- Focus on Data Quality at Source:
- Prioritize data quality at the point of entry to minimize the need for corrections later.
In summary, while there might be exceptional cases where limited historical data changes are necessary, the general principle should be to prioritize the accuracy and consistency of the current record.