Max token and overlap token settings for chunk size
Key Takeaway
The "max token" setting caps how much text goes into each chunk, 512 tokens by default. The "overlap tokens" setting repeats text across chunk breaks. For non-Latin-based languages such as Japanese, set the max token setting lower than 512, because a default-size chunk can go over the embedding model's 512-token limit and lose text from its embedding.
Before Data 360 can search unstructured content, it splits the content into chunks and turns each chunk into a vector. Two settings shape those chunks: max token and overlap tokens. You choose both when you create a search index. Their values directly affect how relevant retrieval results are.
How Data 360 turns content into vectors
When Data 360 builds a search index, it works in three steps:
- Data 360 separates your content into sentences.
- Data 360 merges the sentences into chunks based on the max token setting.
- The embedding algorithm converts each chunk into a vector, so one chunk results in one vector.
Embedding models ignore any text beyond their token limit, so a chunk that is too large is only partly embedded.