Shadow Mode — AI Glossary

Reference details

Related terms

• Canary Deployment
• A/B Testing (LLMs)
• Evaluation Harness

How it works

1. The request router duplicates each incoming request, sending one copy to the production model and one to the candidate.
2. Only the production response is returned, so the candidate's behaviour reaches no user and a bad candidate causes no incident.
3. Both responses are logged against the same request identifier, which is what makes them comparable on identical input.
4. The candidate call is issued asynchronously so its latency does not enter the user-facing path, and its failures are swallowed rather than surfaced.
5. Comparison happens offline, scoring the paired outputs against whatever quality measure the deployment uses — a judge model, a heuristic, or human review of a sample.
6. Side effects must be suppressed in the shadow path, since a candidate that calls tools will otherwise send real emails and write real rows.

Example: Full mirroring costs ten times a sample and proves nothing more

Take 400,000 production requests a day at 1,500 input and 400 output tokens, so 600,000,000 input and 160,000,000 output tokens. At $3 and $15 per million that is 600 x $3 = $1,800 plus 160 x $15 = $2,400, so $4,200 a day. Mirroring every request to a candidate adds another $4,200 a day. Now ask what sample size the comparison actually needs. To detect a 2-percentage-point difference in a quality rate near 50% at conventional 95% confidence and 80% power, the requirement per arm is roughly 16 x p(1-p) / d², which is 16 x 0.25 / 0.0004 = 10,000 paired observations. A 10% shadow sample yields 400,000 x 0.10 = 40,000 paired observations a day, so the needed 10,000 arrive in 10,000 / 40,000 = 0.25 of a day, which is 6 hours. The comparison is therefore between $4,200 a day for full mirroring and $420 a day for a 10% sample that reaches statistical adequacy before lunch — a difference of $3,780 a day, or about $1,379,700 a year, buying no additional confidence in the result. Full mirroring earns its cost only for a different purpose: catching rare failures that a 10% sample would miss, where the question is not whether quality differs on average but whether the candidate breaks on an input type occurring in well under 1% of traffic.

Commonly confused with

Shadow mode is not a canary release, and the difference is who receives the output. A canary sends a small percentage of real users to the new version and serves them its responses, so it tests the candidate in production including its effect on users — and a bad canary causes real harm to that slice. Shadow mode serves no user from the candidate, so it is safe but cannot measure anything downstream of the response: engagement, task completion, or whether users came back. The two are sequential rather than alternative. Shadow mode answers whether the candidate's outputs look acceptable, and a canary answers whether users respond to them acceptably, which is a question no amount of offline comparison reaches.

When to use it

Reach for it when:

• Before a model or prompt change reaches any user, since the candidate's outputs are evaluated on real traffic at zero user risk.
• At a sampled rate rather than full mirroring, as 10% supplied the needed 10,000 paired observations within 6 hours above.
• When real traffic distribution matters, which a curated eval set does not reproduce.
• With side effects suppressed in the shadow path, which is the precondition for running it against tool-using agents at all.

Reach for something else when:

• At 100% mirroring for an average-quality comparison, which cost $3,780 a day more for no additional confidence.
• To measure user-facing outcomes, which require users to receive the output and therefore a canary.
• Without suppressing side effects, where a shadowed agent performs real actions twice.
• As the final gate before full rollout, since it cannot detect how users respond to the candidate's behaviour.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Shadow Mode — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path