OSWorld — AI Glossary

Reference details

Related terms

• Computer-Using Agent (CUA)
• GAIA
• Web Agent

How it works

1. Tasks are posed inside real applications in a real operating system, not in a simulated or text-only environment.
2. The agent observes the screen and emits mouse and keyboard actions, so the interface is the same one a person uses.
3. Completion is verified functionally — by inspecting the resulting file or application state — rather than by comparing a transcript.
4. That verification is what makes the benchmark hard to game: a plausible-looking action sequence that did not produce the state scores zero.
5. Tasks span several applications and many steps, so per-step reliability compounds and long tasks dominate the difficulty.

Example: Why functional verification is the hard part

An agent is asked to edit a spreadsheet and save it. A transcript-based check might accept a sequence that looks correct. Functional verification opens the saved file and inspects the cell, so an agent that clicked the right menus but never committed the change scores zero — correctly. Across a multi-step task at 90% per-step reliability, ten steps succeed 0.9¹⁰ ≈ 35% of the time, which is why reported success rates on real-computer benchmarks sit far below per-step competence.

Commonly confused with

Grouped with web-agent benchmarks, and the environment is broader. A web benchmark constrains the agent to a browser, where the DOM provides structured observation. This spans a desktop — file manager, office suite, editor, browser — so observation is largely pixels and actions are raw input, with no accessibility structure to rely on. It is also not a reasoning benchmark: the difficulty is grounded interaction and multi-step reliability, so a model that reasons well can still score poorly for lack of precise control.

When to use it

Reach for it when:

• Evaluating agents that must operate real applications rather than APIs.
• Functional verification is needed, so plausible-looking action sequences cannot pass.
• Explaining why multi-step grounded tasks are harder than their per-step accuracy suggests.

Reach for something else when:

• The target is browser-only automation, where a web benchmark is closer.
• Measuring reasoning, which this confounds with interface control.
• As a proxy for your own environment, which differs in applications and layout.

Primary source

• Xie et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to OSWorld — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path