How to Build an LLM Gateway: Step-by-Step Guide

Building an LLM gateway means putting a single HTTP service between your applications and one or more model providers, so your apps never talk to a provider SDK directly. That service handles authentication, request routing, streaming, retries, rate limiting, logging, and usage tracking in one place. This guide walks through the actual components you need to build, in the order you need to build them.

If you're evaluating whether to build this yourself or use a hosted one, the short version: a gateway is straightforward to prototype in an afternoon and genuinely hard to run reliably in production. Below is what "hard" actually means, component by component.

Step 1: Define the request/response contract

Before writing any code, decide what your internal API looks like. Most teams converge on something close to the Anthropic Messages API shape, since it's clean and widely understood:

{
  "model": "claude-sonnet-4",
  "messages": [{"role": "user", "content": "Summarize this ticket"}],
  "max_tokens": 1024,
  "stream": false
}

Pick one canonical schema and translate to/from it at the edges. This matters because every consumer of your gateway — internal services, other teams, external partners — should only ever need to learn one contract, not the quirks of whichever provider you're calling behind the scenes.

Step 2: Build the auth layer

Your gateway needs its own API keys, separate from the underlying provider credentials. Never hand out the raw provider key to application code — if it leaks, you lose control instantly and can't distinguish which caller misused it.

Minimum viable auth layer:

• Generate scoped keys per application (e.g. sub_live_... style prefixes so keys are self-identifying in logs)
• Store a hash, not the raw key, in your database
• Attach metadata to each key: owner, environment, rate limit tier
• Reject requests with missing/invalid keys before they touch any provider call

This is also where you enforce per-key rate limits and quotas, which is what stops one misbehaving app from burning your entire provider budget in an hour.

Step 3: Implement the provider call and streaming

The core of the gateway is a thin adapter that forwards the normalized request to the actual model API and translates the response back. Two things make this nontrivial: streaming and tool use.

For streaming, your gateway needs to proxy Server-Sent Events without buffering the whole response — buffering defeats the purpose and adds latency your users will notice immediately:

const upstream = await fetch(providerUrl, {
  method: "POST",
  headers: { "Authorization": `Bearer ${providerKey}` },
  body: JSON.stringify(payload),
});

return new Response(upstream.body, {
  headers: { "Content-Type": "text/event-stream" },
});

For tool use, you need to pass tool definitions through untouched and correctly relay tool_use / tool_result blocks in both directions. This is where a lot of homegrown gateways break, because it's easy to get the happy path working and much harder to handle multi-turn tool loops correctly.

Step 4: Add retries and failover

Provider APIs return 429s and 5xxs more often than you'd like. Your gateway should retry transient failures with exponential backoff, and ideally fail over to a secondary model or region if the primary is degraded:

async function callWithRetry(fn, attempts = 3) {
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn();
    } catch (err) {
      if (err.status < 500 && err.status !== 429) throw err;
      await sleep(2 ** i * 500);
    }
  }
  throw new Error("upstream unavailable after retries");
}

Get the retry conditions wrong and you either hammer a struggling provider harder, or you silently swallow real client errors. Both are bad.

Step 5: Log usage and cost per key

Every request should produce a usage record: input tokens, output tokens, model, latency, and which key made the call. Without this, you can't answer the two questions that come up constantly: "why is our bill higher this month" and "which team is using this feature." Store these records somewhere queryable, not just in raw logs — you'll want to build dashboards on top of them eventually.

Step 6: Handle multi-tenant concerns

If more than one team or app uses the gateway, you need:

• Per-key or per-team spend limits, with hard cutoffs
• Isolation so one tenant's key can't see another tenant's usage data
• A way to rotate or revoke a key without redeploying anything

This is the part that's easy to skip in a prototype and expensive to retrofit later. Design your key and tenant model on day one even if you only have one internal consumer right now.

When to build vs. buy

Building a gateway is worth it if you need very specific routing logic, custom provider mixes, or you're operating at a scale where the marginal engineering cost is trivial. For most teams, the six steps above represent weeks of ongoing maintenance — retry tuning, streaming edge cases, provider API changes — for functionality that's already solved.

SubToAPI is a hosted version of exactly this: it turns your existing Claude access into a clean HTTPS API with application-scoped sub_live_... keys, streaming, tool use support, and per-key usage metadata built in. If you're building step 2 through 5 above yourself, it's worth comparing the time cost against the pricing — Solo starts at €9/month, Team plans are €19/seat, and there's a free trial at signup. The quickstart docs show the full request/response shape if you want to see exactly what you'd be replacing.

Questions

Do I need a gateway if I only use one model provider? Yes, if more than one application or team calls that provider. A gateway still gives you centralized auth, per-app rate limits, and usage tracking even with a single upstream provider — those benefits don't depend on multi-provider routing.

What's the hardest part of building an LLM gateway? Streaming and tool-use proxying, by a wide margin. The basic request-forwarding logic is simple; correctly relaying SSE chunks and multi-turn tool_use/tool_result exchanges without breaking client behavior is where most homegrown gateways accumulate bugs.

Can I add an LLM gateway without changing my existing application code much? Usually yes, if your gateway mirrors the provider's native API shape. Point your existing SDK's base URL at the gateway and swap the API key — see the Messages API docs for an example of a drop-in compatible request format.