Text Splitter — AI Glossary

Reference details

Related terms

• Chunk Overlap
• Document Loader
• Indexing Pipeline

How it works

1. A target chunk size is chosen in the tokeniser's units rather than characters, because the retrieval and generation limits that matter are token limits.
2. The text is divided at the most structural boundary available, trying separators in order — sections, then paragraphs, then sentences — and descending only when a piece still exceeds the target.
3. An overlap is carried between adjacent chunks so a sentence spanning a boundary appears whole in at least one of them.
4. Format-aware splitters respect markup, keeping a table or code block intact instead of cutting it mid-structure.
5. Semantic splitters embed candidate units and cut where consecutive similarity drops, which places boundaries at topic shifts rather than at a fixed length.

Example: What overlap costs, and why it is still worth paying

A 10,000-token document at 500-token chunks with no overlap gives 20 chunks. With 50 tokens of overlap each chunk advances 450, so it takes ceil(10000/450) = 23 chunks — about 15% more to store and embed. The return is that a fact sitting across a boundary is now complete in one of the two neighbours rather than truncated in both. The failure overlap prevents is silent: retrieval returns a chunk that looks relevant and has the answer's second half missing.

Commonly confused with

The component that performs chunking, and the place the strategy actually lives: fixed-size, recursive by separator, or semantic. That distinction matters when debugging, because "our chunking is wrong" is usually a statement about which splitter is configured and with what separators, not about the chunk size everyone reaches for first.

When to use it

Reach for it when:

• Documents exceed the embedding model's input limit and must be divided before indexing.
• Retrieval is returning chunks that are relevant but incomplete, which points at boundary placement rather than at ranking.
• Source documents have structure — headings, tables, code — worth preserving in the chunk.

Reach for something else when:

• Documents already fit the embedding window and splitting only fragments the context.
• The corpus is naturally short units such as FAQ pairs or product records, which are already chunks.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Text Splitter — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path