Tau-Bench — AI Glossary

Reference details

Related terms

• GAIA
• OSWorld
• Tool-Using Agent

How it works

1. Tasks are drawn from service domains with a backing database, so an agent must read and modify real state rather than produce text about doing so.
2. A simulated user supplies the other side of the conversation, which makes the interaction multi-turn and means the agent must elicit information it was not given.
3. A domain policy document states what the agent is and is not permitted to do, and compliance with it is scored separately from task completion.
4. Success is verified against the database's final state rather than against the transcript, so an agent that claims to have made a booking and did not is scored as failing.
5. The headline metric is reliability across repeated trials rather than success on one: the same task is attempted several times and credited only if every attempt succeeds.
6. That choice is the benchmark's distinguishing feature, because a service workflow that works four times in five is not a workflow anyone can deploy.

Example: Why all-trials-succeed is a different measurement entirely

Suppose an agent completes a task correctly 80% of the time. Measured once, it reports 80%. Measured as all-of-k with k = 8 independent attempts, it reports 0.8^8 = 0.168, about 17%. The same agent, the same capability, and two numbers that support opposite conclusions — 80% sounds deployable and 17% plainly is not. The arithmetic is unforgiving as k grows: at k = 4 it is 0.8^4 = 0.410, and even a 95% agent gives 0.95^8 = 0.663. To reach 90% on all of 8 trials, per-attempt reliability must be 0.9^(1/8) = 0.987 — nearly 99%, against the 80% that a single-trial benchmark would have called a good result. This is the right measurement for the setting, because a customer-service workflow is run thousands of times and the variance is the product risk rather than an averaging detail. It also explains why agent benchmarks that report single-trial success overstate deployability so consistently: they are measuring capability where the question is reliability, and the gap between them is exponential in the number of attempts.

Commonly confused with

All-trials-succeed and best-of-k are opposite measurements and are easy to confuse because both involve k attempts. Best-of-k credits a task if any attempt succeeds, which measures reachable capability and is the right frame when a verifier can select the good attempt. All-of-k credits only if every attempt succeeds, which measures consistency and is the right frame when the agent acts autonomously and no one checks. For the same per-attempt rate the two diverge enormously — at 80% and k = 8 they are 0.8^8 = 0.168 against 1 - 0.2^8 = 0.999997 — so reading one as the other is not a small error.

When to use it

Reach for it when:

• Evaluating agents for autonomous service workflows, where consistency rather than peak capability is the deployment question.
• Assessing policy compliance, which this scores separately from task success and most benchmarks do not score at all.
• Where state verification matters, since grading the database rather than the transcript catches agents that report success falsely.
• As a counterweight to single-trial agent benchmarks, which overstate deployability by an exponential factor.

Reach for something else when:

• As a capability ceiling, which best-of-k measures and this deliberately does not.
• For single-turn or text-only tasks, where the simulated user and database state add nothing.
• Comparing results across different trial counts, since the metric's value depends on k and falls steeply with it.
• Outside the covered domains without re-evaluation, since policy compliance is domain-specific by construction.

Primary source

• Yao et al., $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Tau-Bench — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path