Search indexes for chunked Data 360 content
Key Takeaway
Prompt templates and agent actions get results from a search index through a retriever. New search indexes no longer get a default retriever automatically, so you create one in AI Models yourself. A filter on an indexed field limits that retriever to part of the index.
A search index in Data 360 holds your content as small chunks, and each chunk is turned into a vector embedding, a set of numbers that captures meaning. Prompt Builder, AI agents, and Tableau use the index to find the pieces of your data most relevant to a question, including unstructured content such as documents and conversation histories. Data 360 offers two kinds: vector search indexes, which match on meaning, and hybrid search indexes, which also match on keywords. Prompt templates and agent actions reach an index through a retriever, which applies the index to one use case.
What a search index stores
A search index stores chunked and vectorized data so other applications can search it and retrieve from it. You define each index in a data space. You build each index on a data model object (DMO) or an unstructured data model object (UDMO).
Data must be chunked before it can go into a search index. Chunking breaks the data into manageable, semantically meaningful units. The max token limit controls how much text goes into each chunk. Its default is 512. Data 360 supports several chunking strategies.
After chunking, Data 360 turns each chunk into a vector embedding. Vector embeddings measure how close pieces of text are in meaning, so vector search finds semantically similar content, not only exact word matches. For example, vector search understands that "How to log in to my account?" and "How can I sign on?" are similar queries.
Creating a search index configuration also creates two more data model objects. The chunk data model object (CDMO) stores the chunks. The index data model object (IDMO) stores the vector embeddings.
A DMO mapped from one or more external data lake objects (DLOs) can be an index data source only if caching is enabled for all of those external DLOs.