Claude API Batch Processing for Large Datasets

When you need to run Claude over thousands of rows, documents, or records, the naive approach — one request after another in a for loop — is slow, fragile, and expensive to debug when it fails halfway through. Claude API batch processing for large datasets is really a systems problem: you're managing concurrency, rate limits, retries, cost tracking, and partial failures at the same time.

This article walks through a practical pattern for processing large datasets against the Claude API: how to structure the job, how to control concurrency without hitting rate limits, how to handle failures without losing progress, and how to keep track of cost and token usage across the whole run.

Why naive loops don't scale

A simple sequential loop over 50,000 rows has three problems:

1. It's slow. Even at 2 seconds per request, 50,000 requests is 27+ hours sequentially.
2. It's fragile. One network blip or rate-limit error on row 12,000 can kill the whole job if you haven't checkpointed progress.
3. It's hard to cost-control. Without per-request usage tracking, you find out the total spend only after the job finishes.

The fix is to treat the dataset as a queue, process it with bounded concurrency, checkpoint results as you go, and log usage per request so you can catch runaway costs early.

Structuring the batch job

Break the dataset into individual work items, each independent and idempotent — meaning re-running an item twice produces the same result and doesn't duplicate output. This matters because retries are inevitable at scale.

import fs from "fs";

const rows = JSON.parse(fs.readFileSync("dataset.json", "utf8"));
const results = new Map(); // itemId -> result, used as a checkpoint
const CONCURRENCY = 8;
const MAX_RETRIES = 3;

async function processRow(row) {
  const res = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4",
      max_tokens: 500,
      messages: [{ role: "user", content: `Summarize: ${row.text}` }],
    }),
  });

  if (res.status === 429) {
    throw new Error("RATE_LIMITED");
  }
  if (!res.ok) {
    throw new Error(`HTTP_${res.status}`);
  }
  return res.json();
}

Bounded concurrency, not "all at once"

Firing 5,000 requests simultaneously will get most of them rate-limited or timed out. Instead, run a fixed pool of workers pulling from a shared queue:

async function runBatch(items) {
  let index = 0;

  async function worker() {
    while (index < items.length) {
      const i = index++;
      const item = items[i];
      if (results.has(item.id)) continue; // already done, skip

      let attempt = 0;
      while (attempt < MAX_RETRIES) {
        try {
          const result = await processRow(item);
          results.set(item.id, { status: "ok", result });
          break;
        } catch (err) {
          attempt++;
          const backoff = Math.min(1000 * 2 ** attempt, 15000);
          await new Promise(r => setTimeout(r, backoff));
          if (attempt === MAX_RETRIES) {
            results.set(item.id, { status: "failed", error: err.message });
          }
        }
      }
    }
  }

  await Promise.all(Array.from({ length: CONCURRENCY }, worker));
}

await runBatch(rows);
fs.writeFileSync("results.json", JSON.stringify([...results]));

A concurrency of 6–10 workers is a reasonable starting point for most rate limit tiers. Increase it gradually while watching your 429 rate rather than guessing a high number up front.

Checkpointing so failures don't cost you the whole run

Write results.json (or insert into a database) after every item, not just at the end. If the process crashes at item 30,000 of 50,000, you should be able to restart and skip everything already in results. The if (results.has(item.id)) continue line above is what makes the job resumable — without it, a crash means starting over.

For very large datasets (hundreds of thousands of items), persist progress to a database or a append-only log file instead of a single JSON file you rewrite every time — rewriting a multi-GB JSON file on every checkpoint gets slow.

Handling rate limits and retries correctly

Rate limit errors (HTTP 429) and transient server errors (5xx) should be retried with exponential backoff. Everything else — malformed input, content policy rejections, auth errors — should fail fast and get logged, not retried, since retrying won't fix a bad request.

function isRetryable(status) {
  return status === 429 || status >= 500;
}

If you're running large jobs regularly, using an API layer that handles key rotation and consistent usage metadata reduces the amount of retry logic you need to write yourself. SubToAPI issues application-scoped keys (sub_live_...) and returns usage data per request, which makes it straightforward to log tokens and cost alongside each processed item without building that instrumentation from scratch — see the messages docs for the request/response shape.

Tracking cost as you go

Every response includes token usage. Log it per item so you can catch a bad prompt (one that's generating unexpectedly long completions) after 50 items instead of after 50,000.

const usage = data.usage; // { input_tokens, output_tokens }
totalInputTokens += usage.input_tokens;
totalOutputTokens += usage.output_tokens;

Running a sample of 100–200 items first and extrapolating the token totals to your full dataset size gives you a reliable cost estimate before committing to the full run.

Chunking large inputs

If individual items are long documents rather than short rows, check them against the model's context window before sending. Splitting a document into overlapping chunks and summarizing each chunk, then summarizing the summaries, is more reliable than trying to fit everything into one oversized request.

Streaming vs. non-streaming for batch jobs

For batch processing, non-streaming responses are usually simpler — you don't need token-by-token output, you need the final result written to storage. Reserve streaming for interactive use cases where a human is watching the output arrive.

Getting started

If you're building this against SubToAPI, the quickstart covers key creation and your first request, and the messages endpoint docs cover the full request and response schema you'll need for usage tracking. You can test the batch pattern above during the free trial at signup before committing to a plan — see pricing for Solo, Team, and Scale tiers if you're processing datasets across a team.

Questions

What's a safe concurrency level for batch processing with the Claude API? Start at 6–10 concurrent requests and increase gradually while monitoring your 429 rate. Going higher too fast just produces more rate-limit errors and retries, which is often slower than a moderate, steady concurrency.

How do I avoid losing progress if a batch job crashes? Checkpoint results after every item (to a file or database) and check for existing results before reprocessing an item. This makes the job resumable instead of forcing a full restart.

Should I retry every failed request? No. Retry rate limits (429) and server errors (5xx) with exponential backoff. Fail fast on client errors like bad input or auth failures — retrying those wastes time and money without changing the outcome.