How to Build an Internal LLM Gateway

An internal LLM gateway is a single service that sits between your applications and the LLM providers you use, so that every team in your company talks to one internal endpoint instead of directly to Claude, OpenAI, or whichever model vendor you've picked. You build one when more than one team or app needs model access and you need centralized control over keys, spend, logging, and access policy rather than every engineer holding their own provider credentials.

The core problem an internal gateway solves isn't "how do I call an LLM" — that's one HTTP request. It's "how do I let 15 teams call LLMs without losing track of who's spending what, which prompts are going out, and what happens when a provider has an outage." This article walks through what to actually build, in what order, and where it's worth skipping the build entirely.

What an internal LLM gateway needs to do

At minimum, a usable internal gateway handles:

• Authentication — issuing scoped credentials per team or app, not sharing one provider key everywhere
• Request routing — forwarding to the right model/provider, with a stable internal API shape even if the backend changes
• Usage tracking — token counts and cost per team, per app, per key
• Rate limiting — protecting both your budget and the upstream provider's limits
• Logging — enough request/response metadata for debugging and audits, without necessarily storing full prompt bodies if that's a compliance concern
• Streaming support — most product surfaces (chat UIs, assistants) need token-by-token streaming, not just blocking responses

Missing any of these isn't fatal on day one, but you'll be asked for all of them within a quarter once more than two teams are using the gateway.

Step 1: Decide on the request contract

Before writing code, fix the shape of requests and responses your internal teams will call. Keep it close to the provider's native format so you don't have to maintain a translation layer in your head every time you debug something, but strip out things internal consumers shouldn't control directly, like raw API keys.

A reasonable internal contract looks like:

POST /internal/llm/messages
Authorization: Bearer <internal-team-token>
Content-Type: application/json

{
  "model": "claude-sonnet-4",
  "messages": [{ "role": "user", "content": "Summarize this ticket" }],
  "max_tokens": 1024,
  "stream": false
}

The gateway validates the internal token, looks up which provider key maps to that team, forwards the request upstream, and returns the response — adding usage metadata on the way back.

Step 2: Build the auth and key-mapping layer

This is the part that actually matters. Each team or service gets its own internal token, mapped server-side to a real provider credential (or a shared pool, if you want centralized billing). Never let application code hold the upstream provider key directly — that's the whole point of the gateway.

const teamKeyMap = {
  "team-support": process.env.CLAUDE_KEY_SUPPORT,
  "team-growth": process.env.CLAUDE_KEY_GROWTH,
};

function resolveProviderKey(internalToken) {
  const team = verifyAndDecode(internalToken);
  return teamKeyMap[team.id];
}

In practice you'll want this backed by a real store (Postgres, Redis) rather than an in-memory object, plus rotation support so you can revoke a team's access without redeploying.

Step 3: Add usage metering before anything else breaks

Cost visibility is the single most common reason internal gateways get built in the first place, and the most common reason the first version gets rewritten six weeks in. Track, per request: team ID, model, input tokens, output tokens, latency, and whether it errored. Store this in a table you can query — don't rely on provider dashboards, because those are per-account, not per-internal-team.

INSERT INTO llm_usage (team_id, model, input_tokens, output_tokens, latency_ms, status)
VALUES ($1, $2, $3, $4, $5, $6);

This single table answers "which team is burning the budget" and "is this model getting slower" without any extra tooling.

Step 4: Rate limit per team, not just globally

A single team with a buggy retry loop can exhaust your entire provider quota and take down every other team's traffic. Rate limit at the gateway level per internal token, separately from whatever limits the upstream provider enforces. Token-bucket or sliding-window counters in Redis are enough; you don't need anything fancier for an internal tool.

Step 5: Handle streaming properly

If any consumer is a chat interface, you need server-sent events or chunked transfer passed through cleanly from the upstream provider. The common mistake is buffering the whole response server-side before returning it — this kills perceived latency for end users and defeats the purpose of streaming. Proxy the stream chunk by chunk instead of accumulating it.

Build vs. buy

Everything above is maybe 2–3 weeks of focused engineering for a first version, and then ongoing maintenance: handling new model versions, provider API changes, adding tool-use support, keeping the dashboard current. That ongoing cost is usually underestimated.

If what you actually need is a clean HTTPS API with scoped application keys, streaming, tool use, usage metadata, and team seats — without owning the gateway code — SubToAPI does this on top of Claude access you already have. You get sub_live_... keys per app or team, a dashboard for usage and spend, and streaming and tool use out of the box, starting at the Solo plan (€9) with Team (€19/seat) and Scale (€49/seat) tiers for larger orgs. You can start with a free trial at signup, check pricing, or read the quickstart to see the request shape before deciding whether to build your own layer on top of it or in front of it.

Either path — build or buy — the decision point is the same: once more than one team needs model access with any kind of cost or access control, "everyone has their own API key" stops being workable, and you need something in between your apps and the provider.

Questions

Do I need a gateway if only one team uses the LLM? Probably not yet. A single team can usually call the provider directly with one scoped key and basic logging. Build or adopt a gateway once a second team needs access, or once you need per-team cost attribution.

Should the gateway store full prompt and response bodies? Only if you have a clear reason — debugging, compliance, or quality review — and a retention policy. Many teams log metadata (token counts, latency, model, status) and skip full content storage to limit exposure of sensitive data.

Can an internal gateway support multiple LLM providers at once? Yes, and it's a common reason to build one: a consistent internal request contract lets you route to different providers or models per team without each app changing its integration code.