Claude API Kubernetes Scaling Configuration Guide

If you're running a service on Kubernetes that calls the Claude API, the core scaling problem isn't CPU or memory — it's concurrency against an external rate limit. Horizontal Pod Autoscaler (HPA) by default scales on resource metrics, but Claude API calls are I/O-bound: a pod can sit at 5% CPU while holding dozens of open HTTP connections waiting on model responses. Scaling purely on CPU under-provisions during traffic spikes and over-provisions during idle periods.

This article covers how to configure Kubernetes autoscaling specifically for Claude API workloads: which metrics to scale on, how to avoid tripping rate limits with too many pods, and how to structure your deployment so scaling events don't cause dropped requests or duplicate billing.

Why CPU-based HPA fails for LLM API workloads

A typical request to Claude might take 2–15 seconds depending on output length and whether you're streaming. During that window, the pod holding the request uses very little CPU — it's just waiting on a socket. If your HPA target is cpu: 70%, you'll rarely hit that threshold even when every pod is saturated with concurrent requests, meaning the cluster won't scale out until it's already too late (timeouts, 429s, queued requests).

The fix is to scale on a metric that actually reflects load on your API-calling workload: concurrent in-flight requests per pod, or queue depth if you're using a work queue in front of your Claude calls.

Scaling on custom metrics instead of CPU

The most reliable pattern is to expose a Prometheus metric for in-flight requests and use the Prometheus Adapter or KEDA to drive the HPA.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: claude-worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: claude-worker
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: claude_inflight_requests
        target:
          type: AverageValue
          averageValue: "8"

Here each pod is sized to handle roughly 8 concurrent Claude calls before triggering a scale-out. The right number depends on your pod's memory per connection and whether you're streaming responses (which hold connections open longer than non-streaming calls).

If you'd rather not run Prometheus, KEDA supports scaling off a queue length (SQS, RabbitMQ, Redis) directly, which works well if you've decoupled request intake from Claude API calls with a job queue — a pattern worth adopting anyway, since it lets you retry and backpressure without touching pod count.

Rate limits are the real ceiling, not pod count

Scaling out more pods doesn't help if you then hit your provider's rate limit and start getting 429s back. Before tuning HPA thresholds, calculate your actual ceiling:

max_useful_pods = provider_rate_limit_rpm / (requests_per_pod_per_minute)

Set maxReplicas at or below that number. Scaling past it just means more pods competing for the same rate-limit budget, with no throughput gain and more wasted compute.

If rate limits are a recurring bottleneck rather than an edge case, it's worth moving the Claude calls behind an API layer like SubToAPI that issues its own sub_live_... keys per application. That gives you per-key usage visibility across your fleet of pods — useful for figuring out exactly which deployment or namespace is burning through quota — without having to instrument rate-limit tracking yourself. See the quickstart for how key issuance works.

Connection pooling and pod sizing

Each pod making Claude API calls should reuse HTTP connections rather than opening a new TLS handshake per request. In Node.js:

import https from "https";

const agent = new https.Agent({ keepAlive: true, maxSockets: 50 });

async function callClaude(payload) {
  const res = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    agent,
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(payload),
  });
  return res.json();
}

Set maxSockets to match the AverageValue target in your HPA config, so a single pod's connection pool and its scaling trigger point are consistent. Mismatched values are a common cause of pods that look "idle" by the metric but are actually connection-starved.

Graceful shutdown during scale-down

Scale-down events are where in-flight Claude requests get dropped if you're not careful. Configure a preStop hook and a reasonable terminationGracePeriodSeconds so pods finish active streams before Kubernetes kills them:

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: claude-worker
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 5"]

Pair this with a readiness probe that flips to NotReady as soon as SIGTERM is received, so the pod stops accepting new work but keeps serving requests already in flight. For streaming responses specifically, this matters more — a mid-stream SSE connection killed by a hard pod termination will leave the client with a truncated response and no clean error. See streaming for how streamed responses are structured if you're building retry logic on the client side.

A practical scaling checklist

• Scale on in-flight request count or queue depth, not CPU
• Cap maxReplicas based on your actual rate limit ceiling, not arbitrary headroom
• Match connection pool size per pod to your HPA target value
• Use preStop + readiness probes to drain in-flight requests before termination
• Separate request intake (queue) from Claude API calls (workers) so each scales independently
• Monitor 429 rate per pod as an early warning that you're scaling past your real ceiling

Questions

Does Kubernetes have a built-in way to scale on API rate limits? No. HPA scales on resource or custom metrics you provide. You need to expose a metric (in-flight requests, 429 rate, or queue depth) and wire it in via the Prometheus Adapter or KEDA.

Should I use a queue between my app and the Claude API? Yes, if traffic is bursty. A queue decouples request intake from worker scaling, lets you retry failed calls without re-running upstream logic, and gives KEDA a clean metric (queue length) to scale on.

How do I avoid 429s when autoscaling aggressively? Cap maxReplicas based on your actual rate limit budget per minute, and consider centralizing calls through a managed layer with per-key usage tracking, like SubToAPI, so you can see which workload is consuming quota before you hit the limit. Check pricing for plan-level details.