Data cleansing and deduplication before migration
Key Takeaway
Profile the source data first so limited effort goes to the problems that affect the most records. Then cleanse and standardize the data, and merge only confirmed unintentional duplicates, before you populate the load templates.
Data cleansing, also called data scrubbing, fixes or removes incorrect, corrupted, incorrectly formatted, duplicate, inconsistent, or incomplete data. The work includes correcting errors, standardizing inconsistent values, removing or filtering irrelevant data, and addressing conditions that create risk later on. Before you migrate legacy data into Agentforce Financial Services, you clean and deduplicate it. The loaded records are then accurate, relationships stay intact, and users and AI agents can trust what they see.
How to clean data for a migration
Cleansing comes after data profiling and before the load. Profiling shows where issues occur, so you can fix the problems most likely to cause errors first. Clean the data before you enrich it or unify profiles, so you work more efficiently and reduce avoidable risk.
- Start with the business need, such as an upcoming data migration.
- Decide which data to migrate, because you may not need a whole data set.
- Profile the in-scope objects in the source systems to find the issues and their root causes.
- Prioritize the issues that affect the most data and have the biggest impact on reporting, automation, analytics, and AI.
- Document your decisions and align stakeholders. Cleansing decisions often affect several teams, such as service, operations, and data governance, so use the profiling results to explain your decisions and keep these teams aligned.
- Build one template per object.
- Choose the right fix for each issue type.
- Apply the fixes and prove they worked with before-and-after evidence:
- Run data profiling in a sandbox to set a baseline.
- Apply your changes.
- Run data profiling again to confirm the improvements and catch new issues.
- Repeat in production using proper change controls.
- Populate the templates with the cleaned data, then review the templates.
Harbor Point Credit Union is moving member data from a legacy core banking system into Agentforce Financial Services, and the team has two weeks for cleanup. Profiling the in-scope member data shows two problems. About 60% of member records store the state in mixed formats, such as "CA", "Calif." and "California", so reports by region come out wrong. About 2% of records have a birthdate in the wrong format. The state problem affects far more records and has a bigger impact on reporting and automation, so the team standardizes state values first. They fix the birthdates next, run profiling again to confirm both fixes, and only then populate the templates.