Regression Testing (LLM) — AI Glossary

Reference details

Related terms

• Eval (Production Evaluation)
• LLM-as-Judge
• LLMOps

How it works

1. A fixed dataset of cases is held constant, because a changing dataset cannot distinguish a behaviour change from a test change.
2. Each case has an expected outcome or a previously recorded output, so the comparison is against your own prior behaviour rather than against an absolute standard.
3. The suite runs on every prompt, model or pipeline change, which is what makes it a gate rather than a report.
4. Non-deterministic output is handled by comparing on a property — schema validity, a judge score, key facts present — instead of exact text.
5. Failures that turn out to be improvements are promoted into the new baseline, so the suite tracks intended change rather than resisting all change.

Example: Why the threshold has to allow for judge noise

A 200-case suite scored by an LLM judge whose own run-to-run agreement is about 95%. Re-running the identical pipeline therefore moves roughly 10 cases by chance, so a gate set at 'no case may regress' fires on every run and is quickly ignored. Setting it at the aggregate — say no more than a 2-point drop in pass rate — is above that noise floor. The measurement error of the grader sets the smallest regression the suite can actually detect.

Commonly confused with

A regression test asks whether behaviour changed against your own previous output; an evaluation asks how good it is in absolute terms. Both are needed and they answer different questions. The complication unique to this setting is nondeterminism: identical inputs can produce different outputs, so a regression test needs a tolerance or a semantic comparison rather than the exact-match assertion that works for code.

When to use it

Reach for it when:

• Prompts, models or retrieval configuration change often enough that silent regressions are likely.
• Production failures should become permanent test cases rather than anecdotes.
• A deploy gate is needed, not just a periodic quality report.

Reach for something else when:

• Measuring absolute quality against a standard, which is evaluation rather than regression.
• The dataset changes between runs, which makes differences uninterpretable.
• The gate threshold sits below the grader's own noise, producing failures nobody trusts.

Referenced by

This term is used by the following ModelRefs references:

• offline eval stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Regression Testing (LLM) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path