Chunking sources for AI retrieval
Key Takeaway
Data 360 must chunk long content before it can vectorize the content and add it to a search index. Each chunk becomes one vector, so each chunk should fit within the embedding model's token limit and make sense on its own.
Chunking breaks long content, such as knowledge articles, uploaded files, web pages, or call transcripts, into smaller, meaningful pieces called chunks. Data 360 converts each chunk into numbers that capture its meaning and stores them in a search index. A retriever can then pull only the most relevant chunks into a prompt for an LLM. You must chunk data before Data 360 can add it to a search index.
Why long content gets chunked
Chunking solves two problems with long content such as knowledge articles and uploaded files.
First, LLMs have a maximum token limit for the text they can process in a single call, so a long document may not fit. With chunks, a retriever adds only the relevant pieces to a prompt, so the prompt gets useful information without going over the LLM's token limit.
Second, searching long, complex articles and files takes a lot of time and processing power. Small chunks make search faster, improve relevance, and let Data 360 handle very large datasets.
Text, images, and audio files can all be chunked.