Computer Use — AI Glossary

Reference details

Related terms

• AI Agent
• Tool Use
• Vision-Language Model (VLM)

How it works

1. The screenshot sent to the model is normally downscaled from the real display, so the coordinates the model returns are in the scaled frame and must be mapped back before dispatch.
2. That rescaling is a common source of misclicks: a small rounding error in a dense interface lands on the adjacent control, and nothing in the response indicates the target was missed.
3. Actions are dispatched through an operating-system automation layer as synthetic input events, so the application receives them as it would a real pointer or keyboard.
4. A settle delay is required between acting and capturing, because an interface mid-transition photographs as a half-rendered state the model then reasons about as if it were final.
5. Verification has to be read from the next screenshot rather than from the action, since dispatching a click returns successfully whether or not anything responded to it.
6. That makes the loop open at every step: there is no return value from the environment, only a new image, which is why error recovery rather than action selection is where these agents fail.

Example: Every step is a screenshot, and screenshots are expensive

The loop is: capture the screen, decide, act, capture again. Each capture enters the context as image tokens — often on the order of 1,500 for a full screen — so a twenty-step task spends around 20 × 1,500 = 30,000 tokens on screenshots alone, before any reasoning text. Keep the whole visual history and that grows quadratically; keep only the latest frame and the agent forgets what it already tried. That tension, not model quality, is what makes these agents slow and expensive, and it is why production systems use an API wherever one exists and reserve screen control for the applications that offer none.

Commonly confused with

Computer use is a fallback interface, not an upgrade over tool calling. A tool call is a typed, validated request against a known contract; screen control is inference over pixels with no contract at all, so it is more general and strictly less reliable. Browser automation is the narrower case, where the DOM offers structure that a raw desktop does not.

When to use it

Reach for it when:

• Legacy or third-party applications with no API and no other integration path
• Exploratory and one-off automation, where building an integration is not worth it
• With human confirmation on any irreversible action, since misclicks are routine

Reach for something else when:

• Anywhere an API exists — it will be faster, cheaper and verifiable
• High-volume repetitive work, where per-step screenshot cost dominates
• Unattended operation on consequential actions, given error recovery is the weak point

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Computer Use — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path