Claude API Web Scraping Agent Tutorial

What you'll build

This tutorial walks through building a web scraping agent powered by Claude: you give it a URL and a description of the data you want, and it fetches the page, extracts structured fields (prices, names, dates, whatever you define), and returns clean JSON. Unlike a hardcoded scraper with CSS selectors that breaks the moment a site changes its markup, a Claude-based agent reads the rendered text and reasons about which parts match your schema — so it survives layout changes and works across sites with different structures.

The core pattern is: fetch → extract via tool use → validate → repeat for pagination. We'll build this with Claude's tool use feature, a simple HTTP fetcher, and a JSON schema that forces structured output. By the end you'll have a working agent you can point at product listings, job boards, article archives, or directories.

Why use Claude instead of traditional scraping

Traditional scrapers (BeautifulSoup, Cheerio, XPath selectors) are fast and cheap but brittle. Every site redesign means rewriting selectors. Claude-based extraction trades a bit of speed and cost for resilience: you describe the data you want in plain language, and the model finds it regardless of DOM structure. This is especially useful when you're scraping dozens of different sites with no shared layout — writing one selector set per site doesn't scale, but one prompt usually does.

The tradeoff is real: don't use an LLM to scrape a single page you check once a day. Use it when you need robustness across many sources, when sites change frequently, or when the data you want is semi-structured (descriptions, reviews, specs) rather than cleanly tabular.

Step 1: fetch the page content

Claude doesn't browse the web on its own — you fetch the HTML or rendered text and pass it in the message. For static pages, a plain HTTP request is enough:

async function fetchPage(url) {
  const res = await fetch(url, {
    headers: { "User-Agent": "Mozilla/5.0 (scraping-agent)" },
  });
  const html = await res.text();
  return html;
}

For JavaScript-heavy sites you'll need a headless browser (Playwright or Puppeteer) to get the rendered DOM. Strip scripts and styles before sending the content to Claude — you want visible text and structure, not noise that burns tokens:

import * as cheerio from "cheerio";

function cleanHtml(html) {
  const $ = cheerio.load(html);
  $("script, style, nav, footer, svg").remove();
  return $("body").text().replace(/\s+/g, " ").trim();
}

Step 2: define the extraction schema with tool use

Instead of asking Claude to "return JSON" in free text (which it sometimes wraps in markdown or explanatory text), use tool use to force a structured response. Define a tool whose input schema is exactly the shape of data you want extracted:

{
  "name": "extract_products",
  "description": "Extract product listings from page text",
  "input_schema": {
    "type": "object",
    "properties": {
      "products": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "name": { "type": "string" },
            "price": { "type": "number" },
            "in_stock": { "type": "boolean" },
            "url": { "type": "string" }
          },
          "required": ["name", "price"]
        }
      },
      "has_next_page": { "type": "boolean" }
    },
    "required": ["products", "has_next_page"]
  }
}

Claude responds with a tool call matching this schema instead of prose, which you can parse directly without regex or markdown stripping. This is the same mechanism covered in more depth in the tool use docs — the pattern is identical whether you're calling Claude directly or through SubToAPI.

Step 3: send the request

Here's the full request using SubToAPI's Messages endpoint, which is a drop-in HTTPS API over your existing Claude access:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 2048,
    "tools": [{
      "name": "extract_products",
      "description": "Extract product listings from page text",
      "input_schema": { "...": "see schema above" }
    }],
    "tool_choice": { "type": "tool", "name": "extract_products" },
    "messages": [{
      "role": "user",
      "content": "Extract all products from this page text:\n\n'"$PAGE_TEXT"'"
    }]
  }'

Forcing tool_choice to the specific tool guarantees Claude always returns the extraction rather than a conversational reply. Parse tool_use blocks from the response to get your structured products array directly.

Step 4: handle pagination and rate limits

Most scraping targets are paginated. Loop based on the has_next_page field from your schema, and add a delay between requests to avoid hammering the target site:

let url = startUrl;
const allProducts = [];

while (url) {
  const html = await fetchPage(url);
  const text = cleanHtml(html);
  const result = await extractWithClaude(text);

  allProducts.push(...result.products);
  url = result.has_next_page ? getNextPageUrl(url) : null;

  await new Promise((r) => setTimeout(r, 1000));
}

For large scraping runs, stream the model's response instead of waiting for the full completion — this matters more for long-form extraction than short JSON tool calls, but it's worth knowing about if you extend the agent to also summarize or classify pages. See streaming for the implementation details.

Step 5: validate and retry

LLM extraction isn't perfectly deterministic. Validate the returned JSON against your schema with a library like Zod or Ajv, and retry with a stricter prompt if validation fails:

function validateProduct(p) {
  return typeof p.name === "string" && typeof p.price === "number";
}

const valid = result.products.filter(validateProduct);
if (valid.length < result.products.length) {
  console.warn(`Dropped ${result.products.length - valid.length} malformed rows`);
}

Putting it behind a real API key

If you're running this agent in production — scheduled jobs, a dashboard, multiple team members triggering scrapes — you want proper API keys, usage tracking, and rate limit visibility rather than sharing one raw credential. SubToAPI turns your Claude access into application keys (sub_live_...) with per-key usage metadata, so you can see exactly how many tokens each scraping job burns. Get started in a few minutes via the quickstart guide, or check pricing if you're scaling this across a team.

Full reference

• /docs/messages — request/response format for the Messages endpoint
• /docs/tools — full tool use schema reference
• /docs/streaming — streaming responses for long extraction jobs
• /signup — start a free trial

FAQ

Can Claude actually browse websites on its own? No. Claude doesn't have built-in internet access for fetching pages. You fetch the HTML or rendered text yourself (via fetch, Playwright, or Puppeteer) and pass the cleaned text into the message — Claude only handles the extraction and reasoning step.

Is this legal? Scraping legality depends on the target site's terms of service, robots.txt, and local law — it varies by jurisdiction and by what you do with the data. This tutorial covers the technical implementation only; check the terms of any site before scraping it.

Why use tool use instead of asking for JSON in the prompt? Asking for "JSON only" in plain text still risks markdown fences, explanatory preambles, or malformed syntax. Tool use with a strict input_schema and forced tool_choice produces a parseable structured object every time, which is far more reliable for production pipelines.