Designing a Response Caching Layer for the Claude API

Why cache Claude API responses at all

If you're searching for how to design a caching layer for Claude API responses, you're almost certainly trying to solve one of two problems: your API bill is growing faster than your user base, or your app feels slow because every request round-trips to a model that takes seconds to respond. A well-designed caching layer fixes both. It sits between your application and Claude, intercepts requests that have already been answered, and returns a stored response instead of paying for and waiting on a new inference call.

The short answer: build a cache keyed on a normalized hash of the prompt (and any parameters that affect output), store responses with a TTL appropriate to how fast your underlying data changes, and invalidate explicitly when the source data changes rather than relying on TTL alone. The rest of this article walks through the actual design decisions — what to hash, where to store it, how to handle streaming, and when caching is the wrong tool entirely.

What actually varies in a Claude request

Before you can cache anything, you need to know what makes two requests "the same." A Claude request isn't just the user's message — it includes:

• The system prompt
• The full message history (not just the latest turn)
• Model name and version
• Temperature, top_p, max_tokens
• Tool definitions, if tools are enabled
• Any images or documents attached

Two requests are only safe to treat as identical if all of these match exactly. A common mistake is caching on the last user message alone, which silently returns wrong answers once conversation history diverges. The cache key has to represent the entire input that produces the output, not just the part that feels variable.

Building the cache key

A practical approach is to serialize the full request payload deterministically (sorted keys, consistent whitespace) and hash it:

import { createHash } from "crypto";

function buildCacheKey(request) {
  const normalized = JSON.stringify(request, Object.keys(request).sort());
  return createHash("sha256").update(normalized).digest("hex");
}

This gives you a stable key regardless of object property ordering. Store the model version in the key too — if you upgrade from one Claude model to another, you want cache misses, not stale answers from a retired model silently served under a new one.

Where to store cached responses

Three realistic options, depending on scale:

In-memory (per-process) — fastest, zero infrastructure, but doesn't survive restarts and isn't shared across instances. Fine for single-server deployments or low-traffic internal tools.

Redis / Memcached — the default choice for most teams. Shared across instances, supports TTLs natively, and handles the key sizes involved (hashes are fixed-length regardless of prompt size). Use SET key value EX ttl and move on.

Database table with an index on the hash column — makes sense if you want to query cache hit patterns, run analytics on what's being asked repeatedly, or need responses to survive longer than a Redis eviction policy allows.

For most teams, start with Redis. It's the least code, the most battle-tested, and easy to replace later if your access patterns justify something more specific.

TTL vs explicit invalidation

Time-based expiry is easy to implement and wrong more often than people expect. If your Claude calls summarize a support ticket, generate a product description from a database row, or answer a question against a knowledge base, the "correct" answer changes whenever the underlying data changes — not on a fixed schedule.

A more reliable pattern:

1. Set a generous TTL as a safety net (e.g., 24 hours) so nothing lives forever.
2. Invalidate explicitly when the source data changes — on ticket update, on product edit, on document re-index — by deleting the specific cache keys tied to that data.
3. Version your cache keys with a data checksum or updated_at timestamp so stale entries simply stop matching new keys instead of needing active deletion.

That third option is often the cleanest: include a content hash of the source data in what you're hashing for the cache key. When the data changes, the hash changes, the key changes, and the old entry just ages out via TTL without you needing an invalidation event at all.

Streaming responses complicate caching

If you're using streaming for latency reasons, caching gets trickier because there's no single "response" object — there's a sequence of chunks. Two common approaches:

• Cache the fully assembled response, then replay it as a simulated stream (chunk it artificially) when serving from cache. This keeps your client-side streaming logic unchanged and still gives you the cost savings, even though the perceived latency benefit for the first cached hit is smaller than for live streaming.
• Don't cache streaming calls at all, and only cache the subset of requests that don't need streaming (batch jobs, background summarization, non-interactive calls). This is simpler and avoids edge cases around partial failures mid-stream.

If you're building on top of SubToAPI, the same caching layer pattern applies on your side of the integration — SubToAPI handles the streaming and tool-use mechanics against Claude (see /docs/streaming and /docs/tools), while your cache sits in front of your own calls to https://api.subtoapi.app/v1/messages to avoid re-sending identical requests. Using application-scoped sub_live_... keys makes it easy to attribute cache hit rates per app or per team when you're reviewing usage in the dashboard.

What shouldn't be cached

Caching is wrong for anything non-deterministic by design or user-specific in a way that defeats reuse:

• Requests with temperature > 0 where you actually want varied output on repeat calls
• Personalized responses that include the user's name, account state, or live data inline in the prompt (unless that data is part of the cache key)
• Anything involving tool calls that hit live external APIs — the tool result, not just the model's text, needs to be part of what you're caching

A good litmus test: if you'd be upset to get the exact same answer twice, don't cache it.

A minimal reference implementation

async function getCachedOrFetch(request, redis, callClaude) {
  const key = buildCacheKey(request);
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  const response = await callClaude(request);
  await redis.set(key, JSON.stringify(response), "EX", 86400);
  return response;
}

This is intentionally plain. Add metrics (hit rate, bytes saved), add stale-while-revalidate if latency matters more than freshness, and add per-key TTL overrides once you know which request types actually repeat.

questions

Does caching reduce Claude API costs even with low hit rates? Yes, proportionally — a 10% hit rate cuts 10% of inference spend with near-zero infrastructure cost on top of Redis you likely already run.

Should I cache at the application layer or rely on a gateway? Application layer gives you control over invalidation logic tied to your own data changes, which a generic gateway cache usually can't replicate correctly.

Is response caching compatible with usage-based billing per request? Yes — cached responses simply aren't sent to the model, so they don't appear as billed inference calls; just make sure your own usage tracking counts cache hits separately from live calls.