Computer Use in Claude Code (Desktop + CLI)

Source: What’s New — Week 13 (Desktop launch) + What’s New — Week 14 (CLI launch) Type: Product Feature Product: Claude Code (Desktop + CLI) Status: Research preview

Computer use lets Claude Code drive the actual desktop — open native apps, click through GUIs, screenshot results, and verify changes end-to-end. It’s the verification loop for surfaces that have no API: GUI-only apps, proprietary tools, mobile simulators, hardware control panels.

Two surfaces, two weeks apart:

Surface	Released	How to enable
Claude Code Desktop app	Week 13 (March 23–27, 2026)	Settings → toggle “Computer use” + grant OS permissions
Claude Code CLI	Week 14 (March 30 – April 3, 2026)	/mcp → find computer-use → toggle on

Off by default on both. The Desktop variant asks before each action; the CLI variant flows through normal Claude Code permission modes.

What it’s for

Best uses are surfaces that don’t expose an API:

• Native iOS / macOS apps — tap through onboarding, screenshot each step, regression-test UI flows.
• iOS Simulator — verify a code change end-to-end without a separate test harness.
• Proprietary tools — vendor admin consoles, IDE-only configuration UIs, hardware control panels.
• Anything GUI-only — legacy software, drawing apps, system settings panels.

For web apps, prefer Playwright MCP (faster, deterministic, headless-capable). Computer use is the bridge to everything Playwright can’t see.

How it differs from related products

Surface	Where it runs	What it controls
Computer use (CLI/Desktop)	Your machine	Native UI of any app on your desktop
Cowork Dispatch	Your desktop, dispatched from mobile	Same surface as computer use, but commanded from the Claude app on your phone
Managed Agents	Anthropic cloud sandbox	Cloud-only environments — not your local desktop
Routines	Anthropic cloud	Cloud-only environments — connectors and code, not local GUI

Computer use is the only surface that drives your physical desktop’s native UI from a Claude Code session.

Enabling on Desktop (Week 13)

1. Open Claude Code Desktop → Settings.
2. Toggle Computer use on.
3. Grant the OS-level permissions Claude requests (screen recording, keyboard control, mouse control on macOS; equivalents on Windows / Linux).
4. The next time Claude wants to take a UI action, it asks for confirmation. Approve once per action, or per session, or persistently per app.

> Open the iOS simulator, tap through the onboarding flow, and screenshot each step

Enabling on CLI (Week 14)

1. Run /mcp inside any Claude Code session.
2. Find the computer-use server in the list.
3. Toggle it on.
4. Grant OS-level permissions if not already granted (the same ones the Desktop app needs).

> Open the iOS simulator, tap through onboarding, and screenshot each step

The CLI flow respects standard permission modes. In auto mode (Week 13 RP), safe UI inspection (screenshot, accessibility tree read) flows through; clicks and keystrokes get classified per the auto-mode rules.

Limitations

• Research preview. Expect rough edges — flaky UI elements, occasional misclicks, screenshots that miss the relevant region.
• Slow vs API-driven verification. A click + screenshot loop is materially slower than a Playwright run or a curl. Reach for it only when no API exists.
• Surface area is your whole machine. Permission discipline matters — computer use can theoretically open any app, including ones you didn’t intend. Use auto mode or strict prompt-each-action gating early on.
• Per-OS quirks. macOS needs Accessibility + Screen Recording permissions; Windows + Linux ergonomics may differ.

Key Takeaways

• Computer use is the verification loop for surfaces with no API — native apps, iOS simulator, vendor consoles, hardware UIs.
• Two enable paths: Desktop app toggle (Week 13) and CLI /mcp → computer-use (Week 14). Same underlying capability, different entry surface.
• For web apps, Playwright MCP is faster, deterministic, and easier to script. Computer use covers the gap Playwright can’t reach.
• Off by default. Permission discipline is the load-bearing safety boundary — pair with auto mode or per-action prompts in early use.
• Distinct from Cowork Dispatch (commanded from phone, runs on your desktop), Managed Agents (cloud sandbox, no local GUI), and Routines (cloud, connectors only).

Try It

1. iOS Simulator regression — open Simulator.app, run a build, ask Claude to tap through the flow you just changed and screenshot each step. Compare against last week’s screenshots.
2. Vendor admin console smoke test — log into your CRM / observability / billing dashboard, ask Claude to verify a config change end-to-end without you having to navigate.
3. Native macOS app integration — automate a repeating UI workflow in an app you can’t drive any other way (e.g., a proprietary design or audio tool).
4. Hardware panel — drive an embedded device’s GUI control panel from Claude Code to flash a config or run a calibration cycle.

Related

• Week 13 release digest — Desktop computer use launch announcement.
• Week 14 release digest — CLI computer use launch announcement.
• Cowork Dispatch — phone-to-desktop control surface; uses the same underlying computer-use capability.
• Claude Managed Agents — cloud sandbox alternative for headless / API-driven work.
• Claude Code Routines — durable cloud automation; pairs with computer use when local UI is needed inside a routine.
• CLI Reference — --permission-mode matrix for gating computer-use actions.
• Essential MCP Servers — computer-use is registered as an MCP server in CLI mode.
• Plugins and Marketplaces — Playwright MCP is the faster choice for web automation.

Open Questions

• Cross-OS parity: doc emphasizes macOS examples (iOS Simulator, Accessibility/Screen Recording). What’s the Windows + Linux fidelity baseline?
• Failure modes when an app’s UI changes between sessions — does computer use cache element coordinates or always re-inspect?
• Auto-mode classifier behavior on UI actions — what gets blocked vs allowed by default?
• Cost profile vs Playwright for equivalent browser tasks (screenshot loop vs DOM-driven interactions).