AI & ML / Agents & orchestration / 17_computer_use_and_code_agents.md

Computer-use and code agents

Updated 5 interview angles 5 min read source
On this page5
  1. Computer use: the model drives a UI
  2. Code agents: write, run, read the error, repeat
  3. Where they meet production
  4. Related
  5. Interview angle

Computer-use and code agents

Two agent categories that broke out of research in 2025-2026 and are worth being able to discuss, because they are the ones with the widest gap between demo and production.

They share a defining property: the tool is a general-purpose environment rather than a typed function. That is what makes them powerful and what makes them dangerous.

Computer use: the model drives a UI

The model receives a screenshot, decides on an action — click at these coordinates, type this, scroll — and receives the next screenshot.

text 
screenshot ──▶ model ──▶ click / type / key
     ▲                        │
     └────────────────────────┘

Why it exists: the long tail of systems with no API. A legacy ERP, a vendor portal, a desktop application. Traditional RPA scripts break when a button moves; a model that reads the screen adapts.

Why it is hard

Problem Consequence
Latency seconds per step, tens of steps
Cost a screenshot per step, at image rates
Spatial precision clicks land in the wrong place
Non-determinism the same task takes a different path
Verification did it actually work?

The precision issue is the one people underestimate — it is the same weakness covered in Multimodal models. Models are weak at exact coordinates, so a mis-click is a routine event rather than an edge case, and a mis-click in a UI can submit something.

The rule that follows: prefer an API, then a DOM-level browser tool (Playwright driven by selectors the model chooses), and use pixel-level computer use last. Each step down that list is more robust and cheaper.

Gotcha: every screen the model reads is untrusted input. A web page can contain instructions aimed at your agent, and it will read them as readily as the page content. Prompt injection through a rendered page is the live attack here — see Prompt injection.

Run it in a sandboxed VM or container with its own credentials and no access to anything the task does not need. Not because it might be attacked, but because it will also make mistakes on its own.

Code agents: write, run, read the error, repeat

The loop that works surprisingly well, because unlike most agent tasks it has a real verifier:

text 
write ──▶ run tests ──▶ read failure ──▶ revise
                   │
                   └── pass ──▶ done

Tests and a compiler give ground truth that no LLM-as-judge can match. That is the whole reason this category outperforms other agent applications: the feedback signal is objective and cheap.

The implications for anything else you build: look for a verifier. An agent task with a mechanical check — schema validation, a query that runs, a reconciliation that balances — will work far better than one scored by another model.

What decides success

  • Context selection. The repository does not fit in the window, so what gets retrieved decides everything. This is a RAG problem wearing different clothes.
  • A tight test loop. If the suite takes 20 minutes, the agent gets a handful of iterations. Fast, focused tests are an agent-productivity feature.
  • Scope. Well-specified bug fixes and mechanical refactors work. “Design the new billing system” does not.

The benchmark to know by name is SWE-bench, which measures resolving real GitHub issues. Scores rose steeply through 2025-2026; the caveat worth stating is that a benchmark of well-described issues with existing tests flatters the category relative to the vague tickets real backlogs contain.

Where they meet production

For both, the deciding questions are the same and they are not about the model:

  1. What is the blast radius? A code agent with push access to main, or a computer-use agent logged into an admin console, is a privilege decision before it is an AI one.
  2. Who reviews? These belong behind a pull request or an approval step. The output is a proposal.
  3. How do you know it worked? Tests for code. For computer use, a verification step that reads the resulting state rather than trusting the action succeeded.
  4. What does it cost per task? Tens of steps at image rates is a real number, and often the reason a pilot does not scale.

The senior position: these are assistants that produce reviewable artefacts, not autonomous workers. Deploying one with unattended write access to a production system is the answer that ends an interview badly.

Interview angle 5

  • “What is computer use, and when would you reach for it?” - the model reads a screenshot and emits clicks and keystrokes, for the long tail of systems with no API. Prefer an API, then DOM-level browser automation, and pixel-level computer use last — each step down is more robust and cheaper.
  • “Why are code agents better than other agents?” - they have a real verifier. Tests and a compiler give objective, cheap ground truth, which no LLM-as-judge matches. The transferable lesson is to look for a mechanical check in any agent task you design.
  • “What decides whether a code agent succeeds?” - context selection, because the repository does not fit in the window, so it is a retrieval problem. Then test suite speed, because a 20-minute suite caps the number of iterations, and scope — bug fixes and mechanical refactors work, open-ended design does not.
  • “What’s the security model?” - every screen or page is untrusted input, so injection through rendered content is the live attack. Sandbox the environment, give it only the credentials the task needs, and treat the agent as something that will also make ordinary mistakes.
  • “Would you run one unattended?” - no. They produce reviewable artefacts: a pull request, a proposed action behind an approval. Unattended write access to production is a privilege decision that has nothing to do with how good the model is.
Previous
Concurrency and backpressure