Claude API on Kubernetes: Deployment & Scaling Guide

What this guide covers

If you're running a service that calls the Claude API from inside a Kubernetes cluster, you're really solving two separate problems: how to deploy the application that talks to Claude reliably, and how to scale that application without hammering your rate limits or leaking your API key through dozens of pods. This guide walks through both, with working manifests you can adapt.

The short answer: treat your Claude-calling service like any other stateless HTTP backend — deploy it behind a Deployment + Service, manage the API key as a Secret, and scale with a HorizontalPodAutoscaler driven by request latency or queue depth rather than raw CPU. The nuance is in how you handle streaming connections, retries, and concurrency caps so that scaling pods doesn't just mean scaling 429 errors.

Base architecture

A typical setup looks like this:

• A Deployment running your API wrapper (Node.js, Python, Go — whatever calls the Claude API)
• A Service exposing it internally or via an Ingress
• A Secret holding your Claude API key
• A HorizontalPodAutoscaler watching request concurrency or CPU
• Optionally, a queue (Redis, SQS, or an in-cluster message broker) in front of the pods if you process requests asynchronously

Keep the Claude-calling layer thin and stateless. Don't store conversation state in pod memory — if a pod dies mid-scale-down, you don't want to lose context. Push that to a database or cache layer outside the pod lifecycle.

Managing the API key as a Secret

Never bake the key into your image or ConfigMap. Use a Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: claude-api-key
type: Opaque
stringData:
  CLAUDE_API_KEY: "sk-ant-xxxxxxxxxxxxxxxx"

Mount it as an environment variable in your deployment, and rotate it with a tool like External Secrets Operator or Sealed Secrets if you're syncing from a vault. If you have multiple teams or projects sharing one underlying Anthropic account, giving every pod the same raw key makes usage tracking and revocation painful — you can't tell which deployment burned through your quota.

This is one reason teams put a layer like SubToAPI between their cluster and the underlying Claude access: instead of distributing one shared secret across every service, each deployment gets its own scoped sub_live_... key, so a runaway pod or misconfigured retry loop in one namespace doesn't drain budget meant for another project.

Example Deployment manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: claude-worker
spec:
  replicas: 3
  selector:
    matchLabels:
      app: claude-worker
  template:
    metadata:
      labels:
        app: claude-worker
    spec:
      containers:
        - name: claude-worker
          image: registry.example.com/claude-worker:latest
          envFrom:
            - secretRef:
                name: claude-api-key
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 15

Resource limits matter more than usual here: Claude calls can hold open HTTP connections for a while, especially with streaming, so set CPU and memory requests based on actual concurrent connection load, not idle usage.

Scaling: why CPU-based HPA often isn't enough

A pod that's waiting on a Claude API response isn't CPU-bound — it's I/O-bound. If you autoscale purely on CPU, you'll under-provision during traffic spikes because the pods look "idle" while they're actually saturated with open connections.

Better signals to scale on:

• In-flight request count per pod, exposed as a custom metric
• Queue depth, if requests go through a broker before hitting Claude
• p95 latency, if you have a service mesh or APM exposing it

Example HPA using a custom metric via Prometheus Adapter:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: claude-worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: claude-worker
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: inflight_requests
        target:
          type: AverageValue
          averageValue: "10"

Set maxReplicas deliberately. Scaling to 50 pods doesn't help if your Claude rate limit tops out at a fixed number of concurrent requests — you'll just distribute the same 429 errors across more pods. Cap concurrency at the application level too, with a semaphore or connection pool per pod, so the cluster never issues more simultaneous requests than your quota supports.

Handling retries and backpressure

Scaling horizontally doesn't fix rate-limit pressure — it can make it worse if every pod retries independently on a 429. A few practical rules:

• Implement exponential backoff with jitter in the client, not just a fixed retry count
• Use a shared rate limiter (Redis-based token bucket) across pods if you're close to your quota ceiling
• Return fast failures to upstream callers rather than queueing indefinitely inside the pod

If you're already offloading rate limiting and key management to a managed layer, this is simpler: SubToAPI applies limits and streaming at the account level, so your cluster just needs standard HTTP client retry logic instead of building a distributed rate limiter from scratch. See /docs/streaming for how streamed responses behave under load.

Observability

At minimum, instrument:

• Request count and latency per pod
• Claude API error rates by status code (401, 429, 500)
• Token usage per deployment, if you need cost attribution by team or namespace

Expose these via Prometheus and alert on sustained 429 rates — that's a scaling signal that adding more pods won't fix, since it usually means you've hit an account-level ceiling, not a cluster capacity problem.

Getting started quickly

If you don't want to build the auth, rate-limiting, and key-rotation layer yourself before you even get to the Kubernetes part, start with a hosted gateway. SubToAPI turns your existing Claude access into a standard HTTPS API with per-project keys, so each deployment in your cluster can have its own sub_live_... credential, usage metadata, and streaming support out of the box. Check /docs/quickstart for the setup, or /pricing if you're scoping this for a team.

FAQ

Should I use a service mesh for Claude API traffic inside my cluster? Only if you already need one for other reasons (mTLS, retries, observability). For a single Claude-calling service, a well-instrumented HPA and good client-side retry logic solve most scaling problems without the added complexity.

How many replicas should I run for a moderate-traffic app? Start with 2-3 for availability, then let the HPA scale based on in-flight requests or queue depth rather than guessing a fixed number. Cap maxReplicas based on your actual Claude account concurrency limits.

Can I run Claude API calls from a Kubernetes CronJob instead of a long-running Deployment? Yes, for batch or scheduled workloads (nightly summarization, report generation) a CronJob is simpler and cheaper than keeping pods warm — just make sure the job has its own timeout and retry policy, since there's no pod pool to absorb transient failures.