How to Use Claude Computer Use Tool: Setup Guide

Claude's computer use tool lets a Claude model view a virtual screen through screenshots and control it by sending mouse clicks, keyboard input, and scroll commands. Instead of calling a narrow function like get_weather, you give Claude a generic "computer" tool, and it decides which coordinates to click, what to type, and when to take another screenshot to check its progress. This is how you use it in practice: define the tool in your API request, run a loop that executes each action Claude requests, feed back a screenshot after every step, and stop when Claude signals it's done.

Below is a walkthrough of that loop, the tool definition itself, the environment you need to run it safely, and the common mistakes that make it flaky.

What the Computer Use Tool Actually Does

Computer use is a special tool type (not a regular JSON-schema function) that the model has been trained to operate. When enabled, Claude can request actions like:

• screenshot — capture the current state of the screen
• left_click, right_click, double_click at specific x/y coordinates
• type — send keyboard text
• key — send key combinations (Enter, Tab, Ctrl+C, etc.)
• scroll — scroll in a direction at a location
• cursor_position — get the current mouse location

Your job is not to write these actions — Claude decides them. Your job is to provide the environment (a real or virtual display), execute whatever action Claude requests, and return the result (usually a fresh screenshot) as a tool result message.

Setting Up the Tool Definition

In a raw Messages API call, the computer use tool is declared with a type and display dimensions instead of a JSON schema:

curl https://api.anthropic.com/v1/messages \
  -H "content-type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -d '{
    "model": "claude-opus-4-20250514",
    "max_tokens": 1024,
    "tools": [
      {
        "type": "computer_20250124",
        "name": "computer",
        "display_width_px": 1280,
        "display_height_px": 800
      }
    ],
    "messages": [
      {"role": "user", "content": "Open the settings menu and enable dark mode."}
    ]
  }'

Claude replies with a tool_use block containing an action (e.g. "screenshot" or "left_click" with coordinates). You execute that action against a real environment — usually a headless browser, a VM, or a Docker container running a virtual display — and send the outcome back as a tool_result.

The Action Loop

The core pattern is a loop, not a single request/response pair:

1. Send the user's task plus the computer tool definition.
2. Claude responds with an action (often starting with screenshot to see the current state).
3. Your code executes that action in the sandboxed environment.
4. You send a tool_result back containing a new screenshot (as base64 image content).
5. Claude looks at the screenshot, decides the next action, and you repeat.
6. The loop ends when Claude responds with text instead of a tool call, meaning it considers the task complete.

async function runComputerUseLoop(client, task) {
  let messages = [{ role: "user", content: task }];

  while (true) {
    const response = await client.messages.create({
      model: "claude-opus-4-20250514",
      max_tokens: 1024,
      tools: [{
        type: "computer_20250124",
        name: "computer",
        display_width_px: 1280,
        display_height_px: 800
      }],
      messages
    });

    const toolUse = response.content.find(b => b.type === "tool_use");
    if (!toolUse) break; // Claude is done, final text answer returned

    const result = await executeAction(toolUse.input); // your screen automation code
    messages.push({ role: "assistant", content: response.content });
    messages.push({
      role: "user",
      content: [{
        type: "tool_result",
        tool_use_id: toolUse.id,
        content: [{ type: "image", source: { type: "base64", media_type: "image/png", data: result.screenshot } }]
      }]
    });
  }
}

The executeAction function is where the real work happens — it translates Claude's left_click, type, and key requests into calls against something like Playwright, an X11 virtual display, or a remote desktop protocol.

Environment and Safety Requirements

Computer use needs an actual screen to control. Common setups:

• Docker + Xvfb: a headless Linux container running a virtual X display, with a screenshot tool and xdotool for input.
• Browser automation: Playwright or Puppeteer driving a real browser, useful if the task is web-only.
• Isolated VM: for anything touching a real filesystem or installed apps, always run in a disposable VM, never on a machine with production credentials.

Because Claude is clicking and typing autonomously, treat this like running untrusted code: no access to real credentials, no production systems, and a human review step before anything irreversible (payments, deletions, sending messages) executes.

If You Just Need an API, Not a Full Agent

Computer use is powerful but heavy — it requires you to run and maintain a virtual display environment. If your actual goal is simpler (giving your product a Claude-backed API with tool calling, streaming, and usage tracking, without building agent infrastructure), SubToAPI turns your existing Claude access into a standard HTTPS API with sub_live_... keys. You still define and execute regular function-style tools per /docs/tools — computer use itself is a separate, more involved capability you'd run against the underlying Claude API directly, but everything else (streaming responses, usage metadata, team seats) works the same way described in /docs/quickstart.

FAQ

Do I need a real computer or VM to use this tool?

Yes. Claude only sends action requests — you need actual infrastructure (a virtual display, browser, or VM) to execute clicks, keystrokes, and screenshots and send results back.

Can computer use run without a loop?

No. A single request only gets you one action. Real tasks require the request-execute-screenshot-repeat loop until Claude returns plain text instead of a tool call.

Is computer use safe to run against production systems?

No. Treat it like autonomous untrusted code — run it in an isolated sandbox or VM with no access to real credentials, and add a human approval step for any irreversible action.