Document Q&A — AI Glossary

Reference details

Related terms

• RAG (Retrieval-Augmented Generation)
• OCR (Optical Character Recognition)
• Intelligent Document Processing

How it works

1. Text extraction comes first and is the stage that most often determines the ceiling: a layout-aware extractor preserves reading order and table structure, while a naive one interleaves columns into unreadable prose.
2. Scanned pages need optical recognition before any of this, and its per-character confidence is the only early signal that the text being indexed is wrong.
3. Chunking then follows document structure where the structure is recoverable — sections, clauses, table rows — rather than a fixed token stride that cuts wherever it lands.
4. Structure is only recoverable where the extractor emitted it, so a scanned document arrives as an undifferentiated stream and has to be chunked by stride whether that suits it or not.
5. Overlap between adjacent chunks mitigates that at the cost of duplicating content in the index, which makes near-duplicate chunks compete for the same top-k slots.
6. Page and section identifiers are attached to each chunk at index time, because a citation has to resolve to a location in the source and that mapping cannot be reconstructed from the text afterwards.

Example: Retrieval sees under a tenth of the document

A 40-page contract at roughly 500 words a page is about 20,000 words, or near 27,000 tokens. Chunked at 512 tokens with 50 tokens of overlap the stride is 462, giving about 58 chunks. Retrieve the top 5 and you have put 2,560 tokens in front of the model — under 10% of the document. That works when the answer sits in one place. It fails on questions whose answer is assembled from a definition on page 3 and an exception on page 31, and it fails when a clause is split across a chunk boundary so neither half states the whole rule. Both failures look like the model being wrong, and neither is.

Commonly confused with

Document question answering is retrieval-augmented generation over a bounded, user-supplied set — which makes it different from long-context prompting, where the whole document goes in and no retrieval step can lose anything. If a document fits in the window, putting it in the window is usually more reliable than chunking it; retrieval earns its place when the corpus does not fit or the cost does not justify it.

When to use it

Reach for it when:

• Corpora too large for the context window, or too costly to send in full
• Where answers must cite a location in the source document
• Repeated queries over a stable set, where embedding once amortises

Reach for something else when:

• Single documents that fit comfortably in context — chunking only adds failure modes
• Questions requiring the whole document at once: totals, counts, or “does this ever say…”
• Scanned or low-quality sources without checking OCR first — retrieval cannot fix bad text

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Document Q&A — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path