ModelRefs / Chunking — AI Glossary
Chunking — AI Glossary
Splitting source documents into smaller segments before embedding for RAG retrieval. Chunk size affects both retrieval quality and context utilization.
Overview
Chunk size affects both retrieval quality and context utilization. Fixed-size (512–1024 tokens) is simple. Semantic chunking splits on content boundaries for better coherence. Sliding window adds overlap. Parent-document retrieval uses small chunks for search but returns larger context.
Reference details
| Topic | rag |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
How it works
- A size is chosen in the tokeniser's units, setting how much context one retrieval result can carry.
- Boundaries are placed either at a fixed length, at structural separators, or where embedded similarity drops — and that choice is the strategy.
- An overlap decides whether material spanning a boundary survives intact in at least one chunk.
- Each chunk is embedded as an independent unit, which is why a chunk covering several topics produces a vector that matches none of them well.
- Size therefore trades two things against each other: smaller chunks match precisely and may not contain the answer, larger ones contain it and match less sharply.
Example: The trade is in the vector, not the text
A 2,000-token section covering four topics embeds to a single vector that is the average of four things, so it ranks moderately for all four queries and highly for none. Split into four 500-token chunks, each vector represents one topic and ranks sharply for its own query. The text is identical in both cases; only the granularity of the representation changed. That is why chunk size is a retrieval-quality parameter rather than a storage one.
Commonly confused with
The operation; a text splitter is the component that performs it, the way memory relates to a memory store. Chunk size and overlap are its parameters, and they are the two settings most often blamed for RAG quality when the real cause is elsewhere. Worth measuring against retrieval metrics rather than tuned by feel, since the effect of a size change is not visible in the generated answer.
When to use it
Reach for it when:
- Documents exceed the embedding model's input limit and must be divided.
- Retrieval returns results that are topically close but never precisely right, which points at chunks that are too large.
- Tuning the balance between match precision and sufficient context.
Reach for something else when:
- Source units are already short and self-contained, such as FAQ pairs or records.
- Chunking is being tuned when the real problem is upstream — a loader producing garbled text.
- The corpus fits in context and retrieval is not needed at all.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Chunking — AI Glossary.
Frequently asked questions
What is Chunking?
Splitting source documents into smaller segments before embedding for RAG retrieval.
What concepts are related to Chunking?
Closely related concepts include rag, embedding, semantic search.