LlamaIndex — AI Glossary

Reference details

Related terms

• LangChain
• RAG (Retrieval-Augmented Generation)
• Vector Database

How it works

1. Connectors load documents from their sources — files, databases, APIs — into a common document representation, which is the step that makes heterogeneous corpora addressable at all.
2. Documents are parsed into nodes: chunks carrying text plus metadata and relationships to neighbouring and parent nodes. The metadata is what later supports filtered retrieval, and omitting it at ingestion forecloses that permanently.
3. Nodes are embedded and written to a vector store, alongside whatever other index structures are needed — keyword, summary, or a graph over node relationships.
4. A retriever fetches candidates for a query from one or more of those indexes, and a postprocessor stage reranks, filters by metadata, or expands a hit to its surrounding nodes.
5. A query engine assembles the retrieved nodes into a prompt and calls the model, with a response synthesiser deciding how multiple nodes are combined — stuffed into one call, refined across successive calls, or summarised in a tree.
6. Chunking is the decision with the longest reach: it is fixed at ingestion, every retrieval quality property depends on it, and changing it means re-embedding the corpus.

Example: Why the chunk size is the expensive decision to revisit

Take 1,000 PDFs of about 20 pages at roughly 500 words a page: 1,000 x 20 x 500 = 10,000,000 words, call it 13M tokens at 1.3 tokens per word. Chunking at 512 tokens with 50 tokens of overlap gives an effective stride of 512 - 50 = 462, so the corpus becomes about 13,000,000 / 462 = 28,139 nodes. Embedding 13M tokens at 0.02 units per million costs 13 x 0.02 = 0.26 units — negligible, and this is the figure people quote when they say re-indexing is cheap. It is cheap in API spend and not in anything else. Changing to 256-token chunks raises the node count to roughly 13,000,000 / 206 = 63,107, a factor of 63,107 / 28,139 = 2.2, which changes retrieval behaviour across the whole corpus, so every evaluation result gathered against the old chunking is void and the retrieval quality work restarts. The cost of revisiting chunk size is a re-run of the evaluation suite and the judgement built on it, not 0.26 units of embedding — which is the argument for settling chunking against a real evaluation set before ingesting at volume.

Commonly confused with

LlamaIndex and LangChain overlap heavily and are not chosen on features any more. Both load, chunk, embed, retrieve and call models; both have an orchestration story. The honest distinction is emphasis: LlamaIndex's abstractions are built around indexing and retrieval over documents, with node relationships and multiple index types as first-class concerns, while LangChain's centre of gravity is composing components into chains and agents. Either can do the other's work. The decision that matters is whether your retrieval layer is complex enough to want index abstractions, or simple enough that a vector store client and your own code is less to maintain than a framework.

When to use it

Reach for it when:

• Multi-source corpora where connectors save real integration work — mixed file formats, databases, SaaS APIs.
• Retrieval that needs structure: metadata filtering, parent-child expansion, or several index types over one corpus.
• Prototyping a RAG pipeline quickly, where the default chunk-embed-retrieve path gets you to a measurable baseline fast.
• Multi-document synthesis, where the response synthesiser strategies address a problem you would otherwise write yourself.

Reach for something else when:

• A single collection with one chunking strategy and a straightforward query, where a vector store client and fifty lines of your own code is less to understand.
• When you need exact control of the prompt sent to the model, which framework-assembled prompts make harder to see and to pin.
• Before settling chunking against an evaluation set, since the framework makes it easy to ingest at volume and the decision is costly to revisit.
• As a substitute for retrieval evaluation. The framework supplies the pipeline; recall against your own queries is still yours to measure.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LlamaIndex — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path