ModelRefs / Indexing Pipeline — AI Glossary

Indexing Pipeline — AI Glossary

The offline process of loading, cleaning, chunking, embedding, and storing documents into a vector index for later retrieval.

Overview

Indexing pipelines: load documents (document loaders for PDF, HTML, Notion, Google Drive), clean/normalize, split into chunks (text splitter with overlap), encode with embedding model, and upsert into vector store with metadata. Run on schedule or event-trigger (new document). Tools: LlamaIndex Ingestion Pipeline, LangChain document loaders, Unstructured.io.

Reference details

Topic	rag
Last reviewed	2026-06-24

How it works

Documents are loaded and converted to text with metadata, and the quality of this step bounds everything after it.
Text is cleaned and normalised, since boilerplate and navigation chrome embed as content and dilute the vectors.
Chunking divides the text, with size and overlap set deliberately rather than left at defaults.
Each chunk is embedded and upserted with its metadata, which is what later allows filtering and citation.
The pipeline is re-run on a schedule or on change, and a changed embedding model forces a full re-index because old and new vectors are not comparable.

Example: The cost that makes embedding-model choice sticky

A corpus of 200,000 chunks at 400 tokens each is 80 million tokens to embed. Switching embedding model means re-embedding all of it, because vectors from two models do not share a space — a partial migration silently breaks ranking. At a notional £0.02 per million tokens that is £1.60 in compute, which is trivial, but the pipeline run, the index rebuild and the revalidation are not. That asymmetry is why the embedding model is one of the stickiest early decisions in a retrieval system.

Commonly confused with

Often collapsed into retrieval, though the two run at different times and fail differently. This is the offline half: loading, cleaning, chunking, embedding, storing. Retrieval is the online half that queries what this produced. Most retrieval problems that present as ranking failures originate here — a loader that mangled a PDF, chunk boundaries that split answers, or an index missing recent documents — and no amount of query-side tuning reaches them.

When to use it

Reach for it when:

Building or debugging the offline half of a retrieval system.
Diagnosing retrieval failures that query-side changes do not improve.
Planning re-index cost before committing to an embedding model.

Reach for something else when:

Describing query-time behaviour, which is retrieval.
Where documents fit in context and no index is needed.
Tuning the query side while the loader output has never been inspected.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Indexing Pipeline — AI Glossary.

Frequently asked questions

What is Indexing Pipeline?