Large data volume load preparation and performance
Key Takeaway
For large loads, use Bulk API and sort child records by parent ID. Pause rules, triggers, and Agentforce Financial Services rollups during the load. Run Batch Apex and the rollup batch jobs after the load finishes, not during it.
Loading millions of client, household, and financial account records into Agentforce Financial Services is slow when every insert runs sharing calculations, automation, and rollups. Large data volume (LDV) load preparation means deciding what to load, in what order, with which API, and which processing to pause. Done well, loads finish quickly and the data stays accurate and correctly related.
Load lean
A deployment has a large data volume (LDV) when it has tens of thousands of users, tens of millions of records, or hundreds of gigabytes of record storage. As data grows, design and configuration choices can raise or lower load times by several orders of magnitude.
The core strategy is to load lean, in this order:
- Identify the business-critical operations before you move users to Salesforce.
- Identify the smallest data set and configuration those operations need.
- Build the load strategy from that minimum.
- Load as fast as possible, so the scope of synchronization stays small.
Loading less data also makes loads faster. Sync only the data the business use case needs. Use virtualization or mashups to view noncritical external data instead of loading it. For very large data sets that need long-term storage, Bulk API or Batch Apex can move data into big objects. Big objects keep consistent performance from 1 million to 1 billion records.
Lean loading has limits, because some configuration must already exist. Parent records must exist before their master-detail children can load. Record owners must exist as users before their records load. Deferring the role hierarchy brings no benefit, because performance is almost always the same either way.