Claude API Request Queue Management in Node.js

If you're sending more than a handful of Claude API calls per minute from a Node.js app, you'll eventually hit rate limits, timeouts, or overlapping requests that exhaust your concurrency budget. Request queue management solves this by controlling how many calls go out at once, retrying failed ones with backoff, and making sure nothing gets dropped silently when Anthropic (or any upstream provider) throttles you.

The short answer: you need a queue that limits concurrency, respects 429/529 responses with exponential backoff, and separates "fire and forget" jobs from ones that need a response before continuing. Below is a practical implementation you can drop into a Node.js service, plus the tradeoffs of rolling your own versus using a library.

Why you need a queue in the first place

Claude's API (like most LLM APIs) enforces per-minute token and request limits. If your app has bursty traffic — a bulk import job, a webhook that triggers ten summarizations at once, a chatbot with simultaneous users — you'll see:

• 429 Too Many Requests when you exceed rate limits
• 529 Overloaded when the upstream model is under heavy load
• Timeouts on long-running streaming calls that tie up connections

Without a queue, your code either crashes, silently drops requests, or hammers the API with retries that make the problem worse. A queue gives you a single choke point to apply concurrency limits, retry logic, and prioritization.

A minimal in-memory queue

For most single-process Node.js apps, you don't need Redis or a job runner. A small class with a concurrency limit and backoff logic is enough:

class ClaudeQueue {
  constructor({ concurrency = 3, maxRetries = 5 } = {}) {
    this.concurrency = concurrency;
    this.maxRetries = maxRetries;
    this.running = 0;
    this.queue = [];
  }

  add(taskFn) {
    return new Promise((resolve, reject) => {
      this.queue.push({ taskFn, resolve, reject, attempts: 0 });
      this._drain();
    });
  }

  async _drain() {
    if (this.running >= this.concurrency || this.queue.length === 0) return;

    const job = this.queue.shift();
    this.running++;

    try {
      const result = await job.taskFn();
      job.resolve(result);
    } catch (err) {
      if (this._isRetryable(err) && job.attempts < this.maxRetries) {
        job.attempts++;
        const delay = Math.min(1000 * 2 ** job.attempts, 30000);
        setTimeout(() => {
          this.queue.unshift(job);
          this.running--;
          this._drain();
        }, delay);
        return;
      }
      job.reject(err);
    }

    this.running--;
    this._drain();
  }

  _isRetryable(err) {
    const status = err.status || err.response?.status;
    return status === 429 || status === 529 || status >= 500;
  }
}

Usage looks like this:

const queue = new ClaudeQueue({ concurrency: 3 });

async function callClaude(prompt) {
  return queue.add(() =>
    fetch("https://api.anthropic.com/v1/messages", {
      method: "POST",
      headers: {
        "x-api-key": process.env.ANTHROPIC_API_KEY,
        "anthropic-version": "2023-06-01",
        "content-type": "application/json",
      },
      body: JSON.stringify({
        model: "claude-sonnet-4-5",
        max_tokens: 1024,
        messages: [{ role: "user", content: prompt }],
      }),
    }).then((res) => {
      if (!res.ok) {
        const err = new Error("Claude API error");
        err.status = res.status;
        throw err;
      }
      return res.json();
    })
  );
}

This gives you three concurrent requests max, exponential backoff on retryable errors, and a clean await interface for callers. It's enough for scripts, internal tools, and moderate-traffic services.

When in-memory isn't enough

The approach above falls apart if:

• You're running multiple server instances (each has its own queue, so your global concurrency isn't actually bounded)
• You need persistence — if the process restarts, queued jobs disappear
• You want priority levels (user-facing requests before batch jobs)

For those cases, move the queue to Redis-backed job systems like BullMQ or pg-boss. The retry and concurrency logic stays conceptually the same — concurrency limiter, exponential backoff, dead-letter handling for jobs that exhaust retries — but the state lives outside the process.

const { Queue, Worker } = require("bullmq");

const claudeQueue = new Queue("claude-requests", {
  connection: { host: "127.0.0.1", port: 6379 },
});

new Worker(
  "claude-requests",
  async (job) => {
    return callClaudeApi(job.data.prompt);
  },
  {
    connection: { host: "127.0.0.1", port: 6379 },
    concurrency: 5,
    limiter: { max: 50, duration: 60000 }, // 50 jobs per minute
  }
);

BullMQ's built-in limiter option handles the "per-minute" rate limiting for you, and its retry/backoff config maps directly onto the retryable-error pattern above.

A simpler alternative: let the gateway handle it

Building and maintaining queue logic — concurrency limits, backoff curves, dead-letter queues, per-key rate tracking — is a real maintenance cost, especially once you're running Claude access for a team rather than a single script. This is the core problem SubToAPI is built around: it sits between your app and your Claude access, issuing scoped sub_live_... API keys and handling retries, streaming, and usage metadata so your Node.js code just calls one endpoint.

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.SUBTOAPI_KEY}`,
    "content-type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-sonnet-4-5",
    max_tokens: 1024,
    messages: [{ role: "user", content: "Summarize this ticket." }],
  }),
});

You still benefit from writing a concurrency limiter around your own calls — nothing eliminates the need to bound how many requests your app sends simultaneously — but you offload the provider-side rate limit juggling and key management. See the quickstart and messages API docs for details, or check pricing if you're evaluating it for a team.

Practical tips regardless of which approach you pick

• Cap concurrency conservatively first, then raise it. Start at 2–3 concurrent requests and watch your error rate before increasing.
• Distinguish retryable from non-retryable errors. A 400 (bad request) will never succeed on retry — don't waste queue slots on it.
• Log attempts and final failures separately. You want to know if a job succeeded after 3 retries versus failed after exhausting all of them.
• Set a job timeout. A hung streaming connection that never resolves will quietly eat a concurrency slot forever.

FAQ

Do I need a queue for low-traffic apps? If you're making a few Claude API calls per minute from a script or small app, you likely don't need a formal queue — just wrap calls in a retry-with-backoff function. Queues matter once you have concurrent or bursty traffic.

What's the difference between concurrency limiting and rate limiting? Concurrency limiting caps how many requests run at the same time; rate limiting caps how many requests happen per time window (e.g., 50/minute). You usually need both — a queue handles concurrency, and a limiter (like BullMQ's) handles the per-minute cap.

Should I use BullMQ or a simple in-memory queue? Use in-memory for single-process apps and scripts. Use BullMQ or a similar Redis-backed queue once you're running multiple server instances or need jobs to survive a restart.