Document Loader — AI Glossary

Reference details

Related terms

• Indexing Pipeline
• Text Splitter
• Retrieval Pipeline

How it works

1. A source is read in its native format and converted to text plus metadata, before any chunking or embedding exists.
2. Format-specific extraction is where the difficulty lives: a PDF has layout rather than reading order, so columns, tables and headers must be reconstructed rather than read off.
3. Metadata is captured alongside — source, page, section — which is what later makes a citation possible at all.
4. Structure is preserved where the format carries it, so a downstream splitter has separators to cut on instead of guessing.
5. Output quality bounds everything after it: no retrieval or reranking configuration can recover text that was extracted wrongly.

Example: The failure that looks like a retrieval problem

A two-column PDF extracted without layout awareness interleaves the columns, so every line alternates between two unrelated sentences. The text is then chunked, embedded and retrieved perfectly correctly — and every chunk is nonsense. Weeks can go into tuning chunk size, overlap and rerankers because the symptom is bad answers from a RAG pipeline. Reading one extracted chunk reveals it in seconds. Inspecting loader output is the cheapest diagnostic in retrieval and the one most often skipped.

Commonly confused with

Runs before any of this: it reads a source and produces text, where the splitter divides that text. Confusing the two hides the most common RAG failure — a PDF loader that returns garbled column order or drops table structure means the chunks were wrong before chunking began. When retrieval quality is poor, read the loader's raw output before touching chunk size.

When to use it

Reach for it when:

• Ingesting heterogeneous sources where each format needs its own extraction.
• Citations must point at a page or section, which requires metadata captured at load time.
• Diagnosing bad retrieval — read the extracted text before tuning anything downstream.

Reach for something else when:

• Sources are already clean text, where a loader adds a layer for nothing.
• Assuming extraction is lossless; scanned documents need OCR, which is a different step with its own error modes.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Document Loader — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path