Canary Deployment — AI Glossary

Reference details

Related terms

• Blue-Green Deployment
• A/B Testing (LLMs)
• Model Routing

How it works

1. A small share of live traffic is routed to the new version while the rest continues to the current one, which requires per-request routing rather than an environment switch.
2. Assignment is sticky per user or session, so one person does not alternate between versions mid-conversation.
3. Both arms are monitored on the same metrics — error rate, latency, and a quality signal such as a judge score or thumbs rating.
4. Traffic to the new version is increased in stages, each with a hold period long enough to accumulate a readable sample.
5. A rollback is a routing weight change, which is why the canary is reversible in seconds without a redeploy.

Example: Why 5% needs patience to mean anything

A feature serves 20,000 requests a day. A 5% canary sees 1,000 of them. If the base error rate is 1%, the canary expects 10 errors a day, and the standard deviation on a count of 10 is sqrt(10) ≈ 3.2 — so 13 or 7 errors is ordinary noise, and detecting a rise to 2% takes more than one day at that share. The canary is sound; the mistake is promoting after an hour because nothing looked wrong. At 5% a quiet first hour is roughly 42 requests and tells you almost nothing.

Commonly confused with

Both this and blue-green reduce release risk, by opposite mechanisms. A canary shifts a small fraction of live traffic to the new version and needs per-request routing plus metrics good enough to detect a regression in that fraction. Blue-green switches everything at once. So a canary limits blast radius but exposes some users; blue-green exposes none until it exposes all.

When to use it

Reach for it when:

• A model or prompt change could regress quality in ways offline evaluation did not catch.
• Per-request routing and per-arm metrics already exist in the gateway.
• Exposure must be limited while real traffic validates the change.

Reach for something else when:

• Traffic is too low for a small share to produce a readable signal in reasonable time.
• No quality metric exists, leaving only errors and latency — which a regression may not touch.
• The change must be all-or-nothing, such as a schema migration, where blue-green fits better.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Canary Deployment — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path