A/B Testing (LLMs) — AI Glossary

Reference details

Related terms

• Canary Deployment
• Blue-Green Deployment
• Evaluation Harness

How it works

1. A single variable is isolated — model, prompt, temperature, or retrieval configuration — because changing two at once makes the result unattributable.
2. Traffic is split by a stable hash of user or session id, so a given user sees one variant consistently and their behaviour is not averaged across both.
3. An outcome metric is fixed in advance: task completion, thumbs rating, escalation rate, revenue. Choosing it afterwards is how a null result becomes a positive one.
4. The test runs until a pre-committed sample size is reached, rather than until the numbers look favourable, which is what peeking turns into a false positive.
5. The winner ships and the comparison is kept, because LLM outcomes move when the underlying model is updated beneath you.

Example: The sample size decides whether the test can answer anything

A team wants to detect a change in a 40% task-completion rate and considers a 2-percentage-point lift worth shipping. Splitting 10,000 sessions evenly gives 5,000 per variant; the standard error on a 40% rate at n=5,000 is sqrt(0.4 × 0.6 / 5000) ≈ 0.0069, so about 0.7 points, and the difference of two arms carries roughly sqrt(2) × 0.7 ≈ 1.0 point. A 2-point effect sits about two standard errors out and is detectable; a 0.5-point effect is inside the noise and the test cannot distinguish it from zero however the result reads.

Commonly confused with

Confused with offline evaluation, and the two answer different questions. An offline eval scores outputs against fixed references or a judge and tells you which variant is better on your test set. An A/B test puts both in front of real users and tells you which produces better outcomes on live traffic — which can disagree, because users react to latency, tone and format that a reference-based score ignores. It is also not a canary: a canary watches one new version for breakage and rolls back, while an A/B test runs both arms to a planned sample size specifically to compare them.

When to use it

Reach for it when:

• Two variants both look acceptable offline and the decision rests on real user behaviour.
• There is enough traffic to reach the sample size the effect you care about requires.
• The outcome you care about is observable in product telemetry rather than only in a rating.

Reach for something else when:

• Traffic is too thin to resolve the effect size, in which case the test will return noise with a p-value attached.
• One variant is known to be harmful or broken — that is a correctness fix, not an experiment.
• The question is whether output is factually right, which references and graders answer more cheaply than users.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to A/B Testing (LLMs) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path