Claude API Kubernetes Deployment Guide

Deploying a service that calls the Claude API on Kubernetes is mostly a standard containerized app deployment, with a few specifics around secrets management, timeouts, and autoscaling that matter because of how LLM requests behave: long-lived connections, variable latency, and rate limits tied to your API key.

This guide walks through a complete, production-ready setup: containerizing your Claude API client, managing credentials safely, configuring health checks and resource limits, and scaling pods without hitting rate limits or wasting spend.

Why Kubernetes deployments for Claude API need special attention

A typical REST backend returns responses in milliseconds. Claude API calls, especially streaming ones, can hold a connection open for tens of seconds. This changes three things in your Kubernetes config:

• Readiness probes need to tolerate slow startup if your app pre-warms an HTTP client or SDK.
• Pod termination grace periods need to be long enough that in-flight streaming requests aren't cut off during rolling updates.
• Horizontal scaling should be based on concurrent request count or queue depth, not just CPU, since Claude calls are mostly I/O-bound and won't spike CPU usage.

Step 1: Containerize the service

Keep the container minimal. Here's a typical Node.js example calling Claude:

FROM node:20-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
ENV NODE_ENV=production
EXPOSE 3000
CMD ["node", "server.js"]

A minimal server that proxies to Claude (or to SubToAPI's HTTPS gateway) might look like this:

import express from "express";
const app = express();
app.use(express.json());

app.post("/chat", async (req, res) => {
  const response = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${process.env.SUBTOAPI_KEY}`,
    },
    body: JSON.stringify(req.body),
  });
  const data = await response.json();
  res.json(data);
});

app.get("/healthz", (req, res) => res.status(200).send("ok"));
app.listen(3000);

Step 2: Store credentials as Kubernetes Secrets

Never bake API keys into the image or ConfigMaps. Use a Secret:

kubectl create secret generic claude-api-credentials \
  --from-literal=SUBTOAPI_KEY=sub_live_xxxxxxxxxxxx

Reference it in your Deployment manifest:

env:
  - name: SUBTOAPI_KEY
    valueFrom:
      secretKeyRef:
        name: claude-api-credentials
        key: SUBTOAPI_KEY

If you're running multiple environments (staging, production), use separate keys per environment. SubToAPI issues scoped sub_live_... keys per application, so you can rotate or revoke one environment's key without affecting others — useful when a staging cluster gets compromised or decommissioned.

Step 3: Write the Deployment manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: claude-proxy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: claude-proxy
  template:
    metadata:
      labels:
        app: claude-proxy
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: claude-proxy
          image: registry.example.com/claude-proxy:latest
          ports:
            - containerPort: 3000
          envFrom:
            - secretRef:
                name: claude-api-credentials
          readinessProbe:
            httpGet:
              path: /healthz
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /healthz
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 20
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "500m"
              memory: "256Mi"

The 60-second terminationGracePeriodSeconds matters: it gives Kubernetes time to finish in-flight streaming responses before killing the pod during a rolling update, instead of abruptly closing connections mid-stream.

Step 4: Configure a Service and Ingress

apiVersion: v1
kind: Service
metadata:
  name: claude-proxy-svc
spec:
  selector:
    app: claude-proxy
  ports:
    - port: 80
      targetPort: 3000

If you're terminating SSE streams through an Ingress controller (NGINX, for example), disable response buffering so chunks reach the client as they arrive:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"

Without this, NGINX buffers the whole response before forwarding it, which defeats the purpose of streaming and can trigger idle-timeout errors on long-running Claude requests.

Step 5: Autoscale on concurrency, not just CPU

CPU-based HPA (Horizontal Pod Autoscaler) doesn't work well for I/O-bound Claude API calls — your pods can be handling dozens of concurrent requests while barely touching the CPU. Use KEDA or a custom metrics adapter to scale on request concurrency or queue length instead:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: claude-proxy-scaler
spec:
  scaleTargetRef:
    name: claude-proxy
  minReplicaCount: 2
  maxReplicaCount: 10
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: http_requests_in_flight
        threshold: "20"
        query: sum(http_requests_in_flight{app="claude-proxy"})

Step 6: Avoid rate-limit collisions across pods

If every pod shares one upstream API key and you scale to 10 replicas, you can hit concurrent request limits faster than with a single-instance deployment. Options:

• Use a dedicated key per service/environment rather than one shared key across your whole cluster.
• Put a lightweight queue (Redis-backed) in front of the Claude calls to cap total concurrency cluster-wide.
• Monitor 429 responses in your logging stack and alert before they become user-facing errors.

If managing retries, streaming, and per-key usage tracking across a fleet of pods feels like overhead you'd rather not build, SubToAPI exposes Claude as a plain HTTPS API with per-application keys, so each deployment or namespace can have its own key, visible usage metadata, and no separate SDK plumbing to maintain. See the quickstart and streaming docs for the request/response shapes.

Step 7: Observability

At minimum, instrument:

• Request latency (p50/p95/p99) split by streaming vs non-streaming calls.
• Token usage per request, if your upstream returns it in the response body.
• Error rate by status code, especially 429 (rate limit) and 529 (overloaded).

Export these as Prometheus metrics and keep dashboards per namespace if you run staging and production in the same cluster.

Checklist before going to production

• [ ] API keys stored as Secrets, not environment variables baked into images
• [ ] Separate keys per environment
• [ ] Termination grace period long enough for streaming responses to finish
• [ ] Ingress buffering disabled for SSE/streaming routes
• [ ] Autoscaling based on concurrency, not CPU
• [ ] Alerting on 429/529 response codes

Questions

Do I need a service mesh to deploy Claude API workloads on Kubernetes? No. A service mesh is optional and mostly useful for mTLS between internal services or fine-grained traffic shaping. For a straightforward Claude API proxy, a Deployment, Service, and Ingress are sufficient.

How do I handle Claude API rate limits across multiple pods? Either assign each environment/service its own API key, or centralize concurrency control through a queue so the cluster-wide request rate stays under your plan's limit. Also alert on 429 responses so you catch limit breaches quickly.

Can I run Claude API calls in a Kubernetes CronJob? Yes — for batch or scheduled workloads (report generation, nightly summarization), a CronJob with the same Secret-based credential setup works well. Just set backoffLimit and a reasonable activeDeadlineSeconds since LLM calls can occasionally run long.