Data warehouses and data lakes for off-platform data
Key Takeaway
Salesforce Connect uses external objects to query data in an external data warehouse, such as Snowflake, live at runtime. The data stays where it lives, instead of an ETL pipeline copying it into Salesforce.
Many companies keep data in an external data warehouse, such as Snowflake, outside Salesforce. Salesforce can copy that data in with ETL pipelines. It can also query the data live where it lives with Salesforce Connect and external objects, an approach called Data Virtualization. Data can also go the other way: for example, you can export a Salesforce backup into a Snowflake Data Warehouse.
ETL pipelines and Data Virtualization
Many companies keep these kinds of data in an external data warehouse, such as Snowflake:
- Historical transactions
- Financial records
- Usage data
- Operational metrics
Salesforce can reach that data in two ways. It can copy the data in, or it can query the data where it lives.
The traditional way is an ETL pipeline: scheduled jobs that extract data from the source, transform it, and load copies into Salesforce. Copying has drawbacks:
- Copied data is always delayed.
- Each copy of sensitive data adds to compliance scope.
- Pipelines need ongoing monitoring, failure handling and schema drift management.
Data Virtualization works the other way around. Instead of moving data into Salesforce, Salesforce sends the query out to the external system at runtime, with no replication. The data stays in its authoritative source. Users still see it in these places:
- Page layouts
- Related lists
- Reports
- Flows