Create Representative Workload Evaluations — Tutorial

Goal

This tutorial shows how to evaluate an AI system on tasks that resemble the work it will really be asked to do, and how to turn the result into a decision rather than a score.

The gap it closes is specific. Public benchmark results describe performance on a fixed, shared test under someone else's protocol. They are useful for narrowing a field, and they cannot tell you how a system behaves on your documents, your users' phrasing, your edge cases, or your acceptance bar. A representative workload evaluation answers that second question.

The output is not a leaderboard position. It is a comparison table you can defend: what each candidate got right, what it cost, how long it took, how often a human had to intervene, and how it failed when it failed.

Prerequisites

You need a production decision to make — choosing between candidates, deciding whether to ship, or deciding whether a change is an improvement. Evaluation without a decision attached tends to produce numbers nobody uses.

You also need access to realistic tasks, a way to run candidates under identical conditions, and someone who can judge whether an output is acceptable.

Concepts used here, each defined in the ModelRefs glossary: representative workload, evaluation harness, human-in-the-loop, cost per successful task, schema adherence, grounding, and latency percentile.

Architecture: from workload to decision

The pipeline is linear, and every stage is something you should be able to inspect afterwards:

workload definition → task sampling → expected outcomes → system execution → automated checks → human review → failure classification → decision report

Two properties make it trustworthy. Candidates run under identical conditions — same tasks, same prompts, same retrieval results where applicable, same acceptance criteria — so differences are attributable. And every run is recorded, so a surprising aggregate can be traced back to the individual tasks that produced it.

Treat the harness as something you will run repeatedly, not once. Its value compounds when it becomes the thing you re-run after each change.

Step 1. Define the decision and the task families

Write down the decision first, in one sentence: which candidate ships, whether a change is an improvement, or whether quality clears the bar for launch.

Then break the work into task families — groups of tasks that succeed or fail for the same reasons. A support assistant might have lookup, summarisation, multi-document comparison, and escalation. An extraction system might split by document layout or language.

Families matter because aggregate scores hide uneven performance. A system can look acceptable overall while failing one family badly, and it is usually the failing family that determines whether you can ship.

Step 2. Sample tasks that look like the real distribution

Collect real examples where you can, and realistic constructed ones where you cannot.

Cover three bands deliberately: ordinary cases that make up the bulk of traffic, difficult cases that are legitimate but demanding, and failure-prone cases — ambiguous requests, missing information, adversarial phrasing, and inputs the system should refuse or escalate.

Size the set to the decision. A few dozen well-chosen tasks per family, examined closely, will usually inform a decision better than a large set nobody inspects. Grow it as the workload reveals cases you did not anticipate.

Do not build the set exclusively from examples already used to tune prompts or retrieval. Reusing them inflates results in a way that is invisible in the numbers.

Step 3. Define expected outcomes and acceptance criteria

For each task, record what an acceptable result looks like before you run anything.

Some tasks have an exact expected value — a field, a schema, a classification. Many do not, and for those write the acceptance criteria instead: what must be present, what must not be asserted, when the system should decline, and what makes an answer unusable.

Writing criteria in advance is what stops the evaluation from drifting toward whatever the system happens to produce. If criteria are only written after seeing outputs, they tend to describe the output rather than the requirement.

Where a human will judge, give them the rule and a couple of worked examples of pass and fail. Consistency between graders matters more than the precise rubric.

Step 4. Run candidates and record more than quality

Run every candidate over the same tasks with the same settings, and record for each run: the output, whether it met the criteria, latency, token or request volume, retries, and whether a human had to correct or reject it.

Because these systems are non-deterministic, run each task more than once and look at the spread. A candidate that succeeds on four of five attempts is a materially different proposition from one that succeeds every time, and a single run cannot distinguish them.

Keep the raw outputs. Aggregates tell you something moved; the outputs tell you why.

Step 5. Classify failures and compare trade-offs

Group failures by cause rather than counting them. Typical buckets are wrong content, unsupported claims, invalid structure, missing refusal, and operational failures such as timeouts or rate limits.

Severity matters more than frequency. A rare failure that produces a confidently wrong answer in a regulated workflow can outweigh a common cosmetic one, and an average score treats them identically.

Then compare candidates across dimensions together, not one at a time: task success by family, grounding where applicable, schema adherence, citation quality where relevant, human correction effort, rejection and escalation rate, tail latency, cost per successful task, and the severity profile of failures.

Cost per successful task is the denominator that usually changes the conclusion. A cheaper candidate that needs more retries or more human correction can cost more per completed job than a more expensive one.

Step 6. Keep the set alive

A workload evaluation describes the workload as it was when you built it. Both the workload and the systems move.

Add tasks when production surfaces cases the set does not cover, particularly new failure modes. Re-run the set when you change a model, a version, a prompt, or a retrieval configuration — a pinned candidate can still behave differently after a provider-side change.

Record what changed between runs. Comparisons across time are only meaningful when you know which variables moved.

Benchmark evidence versus workload evidence

Public benchmark evidence helps compare performance on defined tests, under a shared protocol, against other systems.

Representative workload evidence shows how a system behaves on tasks similar to the user's real work, under the conditions and acceptance criteria that apply to that work.

Neither alone proves production suitability. Benchmarks are comparable but not specific to you; workload evaluations are specific to you but not comparable to anyone else. Use benchmarks to narrow a field and a workload evaluation to make the decision — and report which one a claim rests on.

Failure modes

- Cherry-picked examples that flatter one candidate. - Evaluation contamination, where the set overlaps material used to tune the system. - An unrepresentative task mix that over-weights easy families. - Missing edge cases, especially refusals and out-of-scope requests. - Inconsistent human grading across candidates or across days. - Comparing candidates under different settings, prompts, or retrieval results. - Ignoring rejected and failed tasks when computing cost and quality. - Optimising for average quality while a critical family quietly regresses.

Risks and limitations

Small samples produce unstable results, and the temptation to read a small difference as an improvement is strong. Prefer per-family inspection over a single headline number.

Evaluator disagreement is real, and it moves scores independently of the system. Check agreement on a sample before trusting close comparisons.

Workloads change, so a set that is not refreshed slowly stops representing the work.

Operational costs are easy to under-count. Retries, human review time, escalation handling, and idle capacity are part of the true cost of a completed task.

Provider and model versions drift, which means a result has a shelf life and should be dated.

Safety and abuse coverage is usually thinner than functional coverage. Unless you deliberately included adversarial and out-of-scope cases, this evaluation says little about them.

Finally, a workload evaluation is evidence for a decision, not proof of production readiness. It describes behaviour on a fixed set at a point in time.

What to compare next

With a workload harness in place, the comparisons that follow are usually:

- Two or more candidate models on the identical set, read per family. - A configuration change — prompt, retrieval, chunking, reranking — against the same baseline. - Deployment paths, where cost shape and tail latency matter as much as quality; the ModelRefs guide on choosing between managed APIs and self-hosted models covers that decision. - Retrieval-specific quality, where the companion tutorial on measuring retrieval recall and grounding isolates retrieval from generation.

The ModelRefs guides on evaluating model quality and interpreting benchmarks cover how to read the resulting evidence without overstating it.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Create Representative Workload Evaluations — Tutorial.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path