Retrievers versus search indexes
Key Takeaway
A search index stores your chunked, vectorized content and is usually general purpose. A retriever applies that index to one use case. Creating a search index no longer creates a default retriever, so build and activate a retriever in AI Models (or use an Agentforce Data Library) before Prompt Builder users can ground prompt templates with it.
A search index is where Data 360 stores your content so it can be searched by meaning. Before storing the content, Data 360 breaks it into small chunks and converts those chunks into numbers (vectors). A retriever is a configured search that runs against a search index for one specific use case. It passes the most relevant results to prompt templates, agents, and flows. Together, search indexes and retrievers power Retrieval Augmented Generation (RAG). RAG grounds LLM responses in your own accurate, current data without retraining or fine-tuning the model.
Search index versus retriever
RAG in Data 360 has two parts: offline preparation and online usage. The search index is built during offline preparation. The retriever then connects that prepared data to prompts at run time.
| Search index | Retriever | |
|---|---|---|
| What it does | Stores chunked and vectorized data that other applications can search and retrieve from | Runs specialized searches on a search index and returns relevant results to agents, prompt templates, flows, and other features |
| Scope | Usually general purpose, for example an index of an entire knowledge base | Applies a multi-purpose index to one particular use case |
| Where it's built | Created and managed in Data 360 | Built in AI Models (formerly Einstein Studio) |
Retrievers are the bridge between search indexes and prompt templates. Einstein Search is the old name for what is now called Retriever.
For example, think of a search index of your entire knowledge base as the whole library. A retriever is a request to pull only certain shelves, such as the articles in one article category. Because the index is multi-purpose, a different retriever can apply the same index to a different use case.