Best Open Source LLM Gateways Compared (2025)
If you're searching for the best open source LLM gateway, you're probably trying to solve one of three problems: you want a single API surface across multiple model providers, you need request logging and rate limiting without building it yourself, or you're trying to avoid vendor lock-in on infrastructure you control. The good news is there are several mature open source projects that do this well. The trade-off you're accepting is operational: you run the servers, you patch the code, and you own the uptime.
This article covers the main open source options, what each is actually built for, and how to decide whether self-hosting is the right call versus using a managed gateway.
What an LLM gateway actually does
Before comparing tools, it helps to be precise about scope. An LLM gateway typically sits between your application and one or more model providers and handles:
- Routing — sending requests to the right model or provider, sometimes with fallback logic
- Authentication — issuing and validating API keys separate from the underlying provider credentials
- Rate limiting and quotas — protecting your provider account and your budget
- Logging and observability — token usage, latency, error rates
- Format normalization — presenting a consistent request/response shape across providers with different APIs
Not every open source project does all of this equally well. Some are routing-first, some are observability-first, and some are closer to a full proxy with a UI.
Litellm
LiteLLM is probably the most widely deployed open source option. It's a Python library and proxy server that translates calls to a huge number of providers (OpenAI, Anthropic, Bedrock, Azure, local models via Ollama, and more) into a single OpenAI-compatible format.
Strengths:
- Broadest provider coverage of any open source gateway
- Drop-in OpenAI SDK compatibility, so migrating existing code is low-effort
- Built-in cost tracking and budget alerts
- Active community and frequent releases
Watch-outs:
- The proxy server needs a database (Postgres) for persistent config, keys, and usage — that's another service to operate
- Enterprise features like SSO and advanced RBAC are gated behind a paid tier
- Performance under high concurrency depends heavily on how you deploy it (workers, caching layer)
If you're already OpenAI-SDK-shaped and need multi-provider routing, LiteLLM is a reasonable default.
Portkey (AI Gateway, open source core)
Portkey ships an open source gateway core with a hosted control plane on top. The self-hosted piece handles routing, caching, and fallbacks; the hosted piece adds a dashboard, guardrails, and analytics.
Strengths:
- Semantic caching and automatic retries out of the box
- Config-driven routing (JSON configs for load balancing across providers)
- Good latency characteristics since the gateway is written for that purpose
Watch-outs:
- The full feature set (guardrails, detailed analytics) pushes you toward the hosted product
- Smaller community than LiteLLM, fewer third-party integrations documented
Bifrost
Bifrost is a newer entrant focused specifically on being a fast, minimal proxy layer, written in Go rather than Python. If your bottleneck is raw gateway latency at scale, this is worth benchmarking against LiteLLM's proxy.
Strengths:
- Lower per-request overhead due to the Go runtime
- Simple deployment model, fewer moving parts than a full LiteLLM proxy stack
Watch-outs:
- Younger project, smaller ecosystem of examples and integrations
- Less built-in tooling for cost analytics compared to LiteLLM or Portkey
Kong AI Gateway / Envoy AI Gateway
If you already run Kong or Envoy for general API traffic, both now ship AI-specific plugins/extensions for LLM routing, token-based rate limiting, and prompt guarding. This is the right choice if your team's expertise is already in API gateway operations and you want LLM traffic to go through the same infrastructure as everything else, rather than adopting a separate AI-specific tool.
How to choose between them
A quick framework:
- Already using OpenAI SDK conventions everywhere? Start with LiteLLM — migration cost is near zero.
- Need low-latency routing at meaningful scale? Benchmark Bifrost and Portkey's core against your actual traffic patterns; don't take marketing benchmarks at face value.
- Already running Kong or Envoy? Use their AI extensions instead of adding a new piece of infrastructure.
- Small team, no dedicated ops capacity? This is where self-hosting starts to cost more than it saves — see below.
When self-hosting isn't the right trade-off
Open source gateways are free to download but not free to run. You're responsible for:
- Database backups and migrations for the proxy's own state
- Patching for security advisories in the gateway itself
- Scaling the proxy layer under load, separate from scaling your application
- Building or configuring dashboards for token usage and cost per team/project
For teams that just want application-level API keys, streaming, and usage metadata without operating another service, a managed layer can be the faster path. SubToAPI turns your existing Claude access into a clean HTTPS API: you get sub_live_... application keys, streaming responses, tool use, and per-key usage metadata in a dashboard, with team seats if you need to split access across a group. There's no proxy to deploy or database to maintain — you get a key from the dashboard and start making requests against /v1/messages.
curl https://api.subtoapi.app/v1/messages \
-H "Authorization: Bearer $SUBTOAPI_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet-4",
"max_tokens": 512,
"messages": [{"role": "user", "content": "Summarize this ticket"}]
}'It's not a multi-provider gateway in the LiteLLM sense — it's focused specifically on giving you a production-grade API surface for Claude without the ops overhead. If your requirement is genuinely multi-provider routing across five different model vendors, an open source gateway is still the right architectural choice. If your requirement is "give my app a stable Claude API with keys, streaming, and usage tracking," it's worth comparing the total cost of running LiteLLM plus Postgres plus monitoring against a Solo plan before you commit engineering time to it.
questions
What's the most popular open source LLM gateway? LiteLLM has the widest adoption due to its broad provider support and OpenAI-compatible interface, which makes migrating existing applications straightforward.
Do open source LLM gateways cost anything to run? The software is free, but you pay in infrastructure (proxy servers, a database for config/usage) and ongoing maintenance — patching, scaling, and monitoring the gateway itself.
When should I use a managed API instead of self-hosting a gateway? When your need is a stable, authenticated API for a single provider like Claude rather than routing across many providers — a managed option like SubToAPI removes the operational overhead entirely.