Claude API Batch Inference for Large Datasets

Running Claude Inference Over Large Datasets

When you need to run Claude against thousands or millions of rows — classifying support tickets, summarizing documents, extracting structured data from scraped pages — the question isn't "how do I call the API" but "how do I call it reliably, fast, and without blowing through rate limits or budget." That's what batch inference at scale actually means: orchestrating many independent Claude requests against a dataset while respecting concurrency limits, handling partial failures, and keeping token costs predictable.

The short answer: you parallelize with bounded concurrency (not unlimited, not one-at-a-time), you chunk large inputs to stay under context limits, you implement retry-with-backoff for rate-limit and transient errors, and you track token usage per request so cost doesn't surprise you at the end of a multi-hour job. Below is a practical approach you can apply whether you're hitting Claude directly or through a proxy like SubToAPI.

Why Naive Loops Fail at Scale

A simple for loop that calls the API once per row works fine for 50 items. At 50,000 items it breaks down for predictable reasons:

• Rate limits: Anthropic and most API layers enforce requests-per-minute and tokens-per-minute caps. A tight sequential loop either wastes time waiting or a parallel loop without limits gets you 429s.
• Transient failures: Network blips, momentary overload, or timeout errors are normal at volume. Without retries, a single failed row can kill a multi-hour job.
• Memory and state: If the process crashes at row 40,000 of 100,000, you need to resume from where you left off, not restart.
• Cost drift: Without per-request token tracking, you only find out the job cost €400 after it's done.

Batch inference design is really about solving these four problems, not about finding a magic "batch endpoint" that does everything for you.

A Practical Batch Pipeline

1. Chunk and normalize your inputs

Before any API calls, split your dataset into units that fit comfortably within Claude's context window. For long documents, this might mean splitting by section or page; for tabular data, it might mean grouping rows into prompts of a fixed size (e.g., 20 rows per request) to reduce the total number of calls.

function chunkArray(items, size) {
  const chunks = [];
  for (let i = 0; i < items.length; i += size) {
    chunks.push(items.slice(i, i + size));
  }
  return chunks;
}

const rowChunks = chunkArray(datasetRows, 20);

Batching rows into a single prompt (when the task allows it) is usually cheaper and faster than one request per row, since you avoid paying the fixed overhead of system prompts and instructions on every call.

2. Bound your concurrency

Run requests in parallel, but cap how many are in flight at once. A concurrency of 5–10 is a reasonable starting point for most accounts; tune it based on the rate limits you're actually hitting.

async function runWithConcurrency(tasks, limit) {
  const results = [];
  let index = 0;

  async function worker() {
    while (index < tasks.length) {
      const current = index++;
      results[current] = await tasks[current]();
    }
  }

  await Promise.all(Array.from({ length: limit }, worker));
  return results;
}

3. Retry on transient and rate-limit errors

Any production batch job needs exponential backoff on 429 and 5xx responses. Don't retry on 400-level errors that indicate a malformed request — those will keep failing.

async function callWithRetry(fn, maxRetries = 5) {
  let attempt = 0;
  while (true) {
    try {
      return await fn();
    } catch (err) {
      const retryable = err.status === 429 || err.status >= 500;
      if (!retryable || attempt >= maxRetries) throw err;
      const delay = Math.min(2 ** attempt * 500, 20000);
      await new Promise((r) => setTimeout(r, delay));
      attempt++;
    }
  }
}

4. Persist progress

For jobs that run for hours, write results to disk or a database as they complete, keyed by input ID, not just to memory. This lets you resume after a crash by filtering out already-processed rows instead of restarting from zero.

5. Track tokens and cost per request

Every Claude response includes usage metadata (input and output token counts). Log it per request so you can project total cost mid-run instead of discovering it at the end. If you're calling through SubToAPI, this usage data is already surfaced in the dashboard per API key, which is useful when a batch job is shared across a team and you want to see who's consuming what without building your own logging layer. See /docs/messages for the response shape.

Example: Batch Classification Call

Here's a minimal example using the Messages endpoint to classify a chunk of support tickets in one request, reducing the per-row call overhead:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "max_tokens": 1024,
    "messages": [{
      "role": "user",
      "content": "Classify each ticket below as billing, bug, or feature_request. Return JSON array in the same order.\n\n1. Cannot cancel subscription\n2. App crashes on upload\n3. Please add dark mode"
    }]
  }'

Wrapping this call in the concurrency + retry pattern above lets you process a 50,000-ticket dataset in manageable batches of 20–30 tickets per request, cutting total call count by an order of magnitude compared to one request per ticket.

When a Proxy Layer Helps

Running batch inference directly against Anthropic's API means managing your own API key rotation, usage tracking, and billing allocation if multiple people or services share the workload. A thin HTTPS layer like SubToAPI gives each service or team member its own sub_live_... key, so a runaway batch job is traceable to a specific key and billed to a specific seat, without changing how you write the retry and concurrency logic above. Pricing starts at Solo for individual use and Team for shared seats — see /pricing. The /docs/quickstart guide covers key setup if you want to point an existing pipeline at it in a few minutes, and streaming support (/docs/streaming) is useful if any part of your batch job also needs to show live progress to a user rather than running fully headless.

Checklist Before You Run a Large Batch Job

• Estimate total tokens and cost on a 1% sample before running the full dataset
• Set concurrency conservatively and increase only after confirming no 429s
• Log per-request status (success, retried, failed) with input IDs
• Persist results incrementally, not just at job completion
• Validate output format (especially JSON) before writing to your database

Questions

Does Claude have a dedicated batch API endpoint? Anthropic offers a Message Batches API for asynchronous, large-scale jobs with discounted pricing, separate from the standard synchronous Messages endpoint. For most real-time or dashboard-driven workloads, the concurrency-and-retry pattern described here works well against the standard endpoint.

How many requests can I run in parallel? It depends on your account's rate limits. Start with a concurrency of 5–10, monitor for 429 responses, and increase gradually. Batching multiple rows into a single prompt reduces the total request count and often matters more than raw concurrency.

How do I keep batch inference costs predictable? Log input/output token usage per request as the job runs, project total cost from a small sample before scaling up, and cap max_tokens per request to avoid unexpectedly long completions on edge-case inputs.