Representative Workload — AI Glossary

Reference details

Related terms

• Evaluation Harness
• Benchmark Contamination
• Eval (Production Evaluation)
• Context Recall

How it works

1. Real traffic is sampled rather than imagined, because the distribution that matters is the one the system will actually receive and it is rarely the one a team would invent.
2. The sample is stratified along the dimensions that change the answer — question type, document format, language, ambiguity, permission scope — so each is present in roughly its production proportion.
3. Cases where the correct behaviour is refusal are included deliberately, since a set containing only answerable questions cannot measure abstention at all.
4. Expected outcomes are labelled once and the set is then frozen, which is what makes a later comparison between candidates a comparison rather than two separate measurements.
5. Candidates are run against it with retrieval, prompts and parameters held identical, so the only thing varying is the thing under test.
6. Each candidate is run several times, because a single pass on a stochastic system is one sample from a distribution rather than a measurement of it.

Example: The failure mode a benchmark set cannot see

Suppose 15% of real questions have no answer in the corpus — a normal proportion for an internal knowledge base. On 1,000 questions that is 150 unanswerable ones, and if the system correctly declines 40% of them it refuses 60 and answers 90 regardless. Those 90 are confident wrong answers, 9.0% of all traffic, and a benchmark set containing no unanswerable questions reports exactly 0% of this failure mode while scoring the system on the other 850. The hardest part is that measuring abstention well needs a much larger set than measuring accuracy. A 300-question set at a 15% unanswerable rate contains 45 such items, and a 40% abstention rate estimated from 45 observations carries a 95% interval of about 1.96 x sqrt(0.4 x 0.6 / 45) ≈ 0.143, so plus or minus 14.3 points — far too coarse to tell two candidates apart. Resolving abstention to plus or minus 5 points needs roughly 1.96^2 x 0.24 / 0.05^2 ≈ 368.8, meaning 369 unanswerable items, which at a 15% rate implies a set of 369 / 0.15 = 2,460 questions. Set that against what accuracy alone needs. Estimating an overall accuracy rate to the same plus or minus 5 points, at the variance-maximising rate of 0.5, requires 1.96^2 x 0.25 / 0.05^2 ≈ 384.2, so about 385 questions. The two requirements are 2,460 against 385, a factor of about 6.4 — so a team wanting to compare candidates on refusal behaviour, rather than merely confirm it exists, needs an evaluation set several times larger than the one that settles accuracy.

Commonly confused with

A representative workload is not a benchmark, and the distinction is what the measurement generalises to. A benchmark is a fixed public protocol whose value is comparability across systems and papers; a representative workload is private and whose value is predicting behaviour on your traffic. A model can lead a leaderboard and fail on your document layout, your jargon, or your refusal requirements, and neither measurement is wrong — they answer different questions. It is also not a regression suite, though the same set often serves both. A regression suite is tuned to catch known breakages and is biased toward previously found bugs; a representative workload has to preserve production proportions, which means keeping the boring majority of easy cases in it even though they rarely reveal anything.

When to use it

Reach for it when:

• Before committing to a model, where the question is behaviour on your data rather than on a public protocol.
• With unanswerable cases included in production proportion, since their absence reports 0% of a failure mode worth 9.0% of traffic above.
• Sized for the hardest thing being measured, as abstention needed roughly eight times the set that accuracy needed.
• Frozen across candidates, with retrieval and prompts held identical so only the candidate varies.

Reach for something else when:

• Assembled from easy or interesting cases, which distorts the proportions the measurement depends on.
• Sized by convenience, where 45 unanswerable items gave an abstention estimate of plus or minus 14.3 points.
• Allowed to drift mid-comparison, which turns one comparison into two unrelated measurements.
• Read as a single run, since one pass on a stochastic system is a sample rather than a result.

Where this appears on ModelRefs

• create representative workload evaluations

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Representative Workload — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path