LLM Gateway Open Source: Options and Tradeoffs

Searching for an "llm gateway open source" solution usually means one of two things: you want a self-hosted proxy that unifies multiple model providers behind one API, or you're evaluating whether to build that layer yourself versus paying for a hosted version. Both paths are valid depending on your team size, security requirements, and how much infrastructure you want to own.

This article covers what open source LLM gateways actually do, the main projects worth evaluating, the real tradeoffs of self-hosting, and when a hosted gateway is the more pragmatic choice.

What an LLM Gateway Actually Does

An LLM gateway sits between your application and one or more model providers (OpenAI, Anthropic, Google, local models via Ollama, etc.). Instead of your app code calling each provider's SDK directly, it calls one consistent endpoint. The gateway handles:

• Routing requests to the right provider/model
• API key management so provider credentials never touch client code
• Rate limiting and retries to smooth out provider-side throttling
• Usage logging for cost tracking and auditing
• Format normalization so you can swap models without rewriting request/response parsing
• Streaming passthrough for token-by-token responses

Open source versions of this pattern let you run the proxy on your own infrastructure — a container, a serverless function, or a dedicated service — with full control over the code.

Popular Open Source LLM Gateway Projects

A few projects dominate this space. None of these are affiliated with SubToAPI; they're independent tools worth knowing about if you're comparing options.

• LiteLLM — a Python library and proxy server that maps dozens of provider APIs to an OpenAI-compatible schema. Popular for teams that want a single format across providers and are comfortable running a Python service.
• Portkey's open source gateway — a lightweight routing layer with fallback and caching logic, designed to be embedded or self-hosted.
• Kong AI Gateway — an extension of the Kong API gateway with plugins for LLM traffic, useful if you're already running Kong for other APIs.
• LangChain's gateway tooling — less a standalone gateway and more a set of abstractions, but often used as the routing layer inside larger LangChain-based systems.

Evaluate these on three axes: how actively maintained the project is, how well it supports the specific provider features you need (tool use, extended context, streaming), and how much operational overhead you're willing to accept.

The Real Tradeoffs of Self-Hosting

Open source gives you full control, but control has a cost. Before committing to a self-hosted gateway, weigh these factors honestly:

Maintenance burden. Provider APIs change. When Anthropic or OpenAI update their request schema, add a new tool-use format, or deprecate an endpoint, someone on your team has to update the gateway, test it, and redeploy. With a small team, this becomes a recurring tax on engineering time.

Uptime is your responsibility. If the gateway process crashes at 2am, your production app goes down with it. You need monitoring, alerting, and someone on call — the same operational discipline you'd apply to any critical service.

Security surface. The gateway holds your provider API keys. Self-hosting means you own key rotation, secrets management, and patching any vulnerabilities in the gateway code or its dependencies.

Multi-tenant features take real engineering time. Per-application keys, per-team rate limits, usage dashboards, and seat-based access control are all things you'd need to build on top of an open source base — they don't come for free.

None of this means self-hosting is wrong. If you have infrastructure engineers, need to keep all traffic inside your own VPC for compliance reasons, or want a gateway that spans many providers at scale, an open source deployment can be the right call.

When a Hosted Gateway Makes More Sense

If your primary need is turning an existing Claude subscription or API access into a clean, stable HTTPS endpoint for your product — without maintaining a proxy service yourself — a hosted option removes the operational overhead entirely.

SubToAPI does exactly that for Claude specifically: it converts your existing Claude access into an API with per-application keys (sub_live_...), streaming, tool use, and usage metadata, all managed from one dashboard. There's no server to deploy or patch — you generate a key and start making requests.

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-3-5-sonnet-20241022",
    "max_tokens": 1024,
    "messages": [
      {"role": "user", "content": "Summarize the tradeoffs of self-hosting an LLM gateway."}
    ]
  }'

Team plans add per-seat API keys so each developer or environment gets its own credential without sharing a single key across the org. Pricing is straightforward: Solo at €9, Team at €19/seat, and Scale at €49/seat, with a free trial at signup. See /pricing for the full breakdown.

If you're weighing open source versus hosted, a reasonable approach is to start with a hosted gateway to validate your product quickly, then revisit self-hosting later if your scale or compliance requirements demand it. Check the /docs/quickstart guide if you want to see how fast that first path can move.

Choosing Between the Two

A quick decision framework:

• Choose open source if you need multi-provider routing across many vendors, have infrastructure engineers on staff, and require all traffic to stay inside your own network.
• Choose hosted if you're building on Claude specifically, want per-app keys and usage tracking without building it yourself, and would rather ship product features than maintain a proxy.

Both are legitimate architectures — the right one depends on what you're optimizing for: control or speed to production.

FAQ

Is an LLM gateway the same as an API gateway? Not exactly. A general API gateway handles routing, auth, and rate limiting for any HTTP service. An LLM gateway adds LLM-specific concerns: streaming token responses, provider-specific request formats, tool-use schemas, and token-based usage tracking.

Can I self-host a gateway and still use a service like SubToAPI? Yes — they solve different layers. You could run an open source gateway for multi-provider routing and use SubToAPI specifically for turning Claude access into a managed API with per-app keys, or use one or the other depending on which provider you're targeting.

Does open source mean free to run? The software is free, but running it isn't — you still pay for compute, monitoring, and the engineering time to maintain and secure it. Factor that into any build-vs-buy comparison rather than comparing license cost alone.