Open Source LLM API Gateway Options Compared

If you're searching for open source LLM API gateway options, you're probably trying to solve one of two problems: routing requests across multiple model providers through a single interface, or adding a control layer (auth, logging, rate limits, cost tracking) in front of an LLM API you already use. Both are solvable with self-hosted software, and this article covers the actual options, what each one is good at, and where self-hosting stops making sense.

The short answer: LiteLLM, Portkey's open source gateway, BricksLLM, and Kong AI Gateway are the most mature open source projects in this space right now. Each takes a different approach — some are lightweight proxies you drop in front of an existing API, others are full platforms with a UI and database. Which one fits depends on whether you need multi-provider routing, per-key usage tracking, or just a thin reverse proxy.

What an LLM API gateway actually does

Before comparing tools, it helps to be precise about what "gateway" means here, since the term gets used loosely:

• Request routing — forwarding calls to one or more upstream model providers (OpenAI, Anthropic, local models) behind a unified endpoint.
• Key management — issuing your own API keys to internal apps or customers instead of handing out your provider's raw key.
• Usage metadata — logging token counts, latency, and cost per request, per key, or per user.
• Rate limiting and quotas — capping usage per key or per team.
• Fallback and retries — switching providers or models if one fails or hits a rate limit.

Not every project does all of this. Some are pure routers with no persistence; others store logs and keys in a database and need a Postgres/Redis instance to run.

The main open source options

LiteLLM

LiteLLM is the most widely adopted option for normalizing calls across providers. It exposes an OpenAI-compatible /chat/completions endpoint and translates requests to whichever backend you configure — Anthropic, OpenAI, Azure, Bedrock, local models via Ollama, and dozens more. It ships as both a Python SDK and a standalone proxy server (litellm-proxy) with a config file for routing rules, fallbacks, and budgets.

Good fit if: you need one API shape across many providers and don't mind running and patching the proxy yourself.

Watch out for: the proxy needs a database for persistent key/budget tracking, and keeping up with provider API changes is on you.

Portkey (open source gateway)

Portkey's gateway component is open source and focuses on request routing with built-in retries, fallbacks, and semantic caching. It's a single Node.js binary or Docker image, which makes it easier to self-host than heavier platforms. The hosted Portkey product adds observability and a UI on top, but the core routing gateway can run standalone.

Good fit if: you want fallback/retry logic and load balancing across providers without deploying a full backend service.

BricksLLM

BricksLLM is a lighter-weight Go proxy aimed specifically at issuing scoped API keys with spend limits and rate limits in front of OpenAI-style APIs. It's a good option if your main need is "give internal teams or customers their own key with a hard budget cap" rather than multi-provider routing.

Kong AI Gateway

If you're already running Kong for API management, its AI Gateway plugins add LLM-specific routing, prompt templating, and semantic caching on top of Kong's existing plugin ecosystem (auth, rate limiting, logging). This is the heaviest option operationally — it makes sense mainly if Kong is already part of your infrastructure.

Roll-your-own with a reverse proxy

For simple cases, some teams just put an existing reverse proxy (nginx, Caddy, or a small Express/Fastify service) in front of a provider API to inject headers, log requests, and enforce basic rate limits. This isn't a "gateway project" per se, but it's a valid option when your needs are narrow: one provider, a handful of internal keys, no need for multi-provider routing.

// minimal Express proxy example — logging + key injection only
app.post('/v1/messages', async (req, res) => {
  const upstream = await fetch('https://api.anthropic.com/v1/messages', {
    method: 'POST',
    headers: {
      'x-api-key': process.env.ANTHROPIC_KEY,
      'anthropic-version': '2023-06-01',
      'content-type': 'application/json',
    },
    body: JSON.stringify(req.body),
  });
  logUsage(req.headers['x-internal-key'], upstream.headers);
  res.status(upstream.status);
  upstream.body.pipe(res);
});

This works fine until you need streaming edge cases, retries, per-key billing, or team seats handled correctly — at which point you're rebuilding a gateway from scratch.

The real cost of self-hosting

Open source gateways are free to download but not free to run. Budget for:

• A database (Postgres/Redis) for keys, budgets, and logs
• Monitoring and alerting for the proxy itself, separate from your app
• Patching for provider API changes (new headers, new streaming formats, deprecated endpoints)
• Handling SSE/streaming correctly under load, which is where a surprising number of proxy bugs live
• Building a dashboard if you want non-engineers to see usage and manage keys

For a side project or a single internal tool, this overhead is manageable. For a product with paying customers or multiple teams needing seats, budgets, and a usage dashboard, it adds up fast — and it's exactly the gap a managed option fills.

If you'd rather skip the operational side entirely, SubToAPI gives you the same core outcome — your own sub_live_... API keys, streaming, tool use, and per-key usage metadata — as a hosted service on top of your existing Claude access, with team seats built in. Plans start at €9/month with a free trial at signup, and the quickstart gets you a working key in a few minutes. See pricing for the full breakdown across Solo, Team, and Scale.

questions

Is LiteLLM good enough for production? Yes, many teams run it in production, but plan for the operational overhead: a database for persistent keys/budgets, monitoring, and staying current with provider API changes. It's a strong choice if you need multi-provider routing and are comfortable maintaining infrastructure.

Do I need a gateway if I only use one provider? Not necessarily a full gateway — but you'll likely still want key scoping, usage tracking, and rate limiting, which either a lightweight proxy (like BricksLLM) or a managed layer like SubToAPI provides without needing multi-provider routing.

What's the difference between an open source gateway and a managed one like SubToAPI? An open source gateway gives you the software; you run the servers, database, and updates. A managed option handles hosting, uptime, and dashboard features (usage metadata, team seats, streaming, tool use) so you only integrate against an API — see the docs for the exact request/response shape.