LLM API Gateway Open Source: Options and Trade-offs

An LLM API gateway sits between your application and one or more model providers, handling authentication, routing, rate limiting, logging, and sometimes caching or fallback logic. When people search for an "open source LLM API gateway," they're usually trying to solve one of two problems: they want a unified interface across multiple providers (OpenAI, Anthropic, local models), or they want to stop hardcoding provider API keys into every service and centralize access control.

The short answer: yes, there are solid open source options — LiteLLM, Portkey's gateway, and Helicone are the most common — and they're worth using if you're willing to run and maintain the infrastructure yourself. If you're specifically working with Claude and want application-level API keys, usage metadata, and team seats without running a server, a managed layer like SubToAPI solves the same problem with none of the ops overhead. The rest of this article breaks down what these gateways actually do, the main open source projects, and how to decide which approach fits your situation.

What an LLM API Gateway Actually Does

Strip away the marketing and a gateway is doing a handful of concrete things:

• Key management — storing provider credentials server-side so client apps never see them
• Routing — sending requests to different models/providers based on rules (cost, latency, availability)
• Rate limiting and quotas — capping usage per user, team, or API key
• Logging and observability — capturing requests, responses, token counts, latency
• Streaming support — proxying server-sent events without buffering the whole response
• Fallback/retry logic — switching providers or models if one fails or hits a rate limit

Not every gateway does all of this well. Some (like LiteLLM) focus heavily on multi-provider routing. Others (like Helicone) are closer to an observability layer that sits in front of your existing API calls. Knowing which of these six things you actually need will save you from adopting a tool that's solving a problem you don't have.

The Main Open Source Projects

LiteLLM

LiteLLM is the most widely adopted option for normalizing requests across providers into an OpenAI-compatible format. It supports a proxy server mode with virtual keys, budgets, and routing rules defined in a YAML config. It's a good fit if your main pain point is "I call five different model APIs and want one interface."

litellm --model claude-3-opus-20240229 --port 8000

The trade-off is operational: you're running a stateful service, managing its config, patching it, and scaling it yourself. For teams with existing platform engineering capacity, that's a reasonable cost. For smaller teams, it's often more infrastructure than the problem justifies.

Portkey Gateway

Portkey's open source gateway focuses on reliability features — retries, fallbacks, load balancing across API keys — with a lightweight Node.js core. It's a good choice if your priority is resilience rather than multi-provider abstraction.

Helicone

Helicone is primarily a logging and observability proxy. You route requests through it and get dashboards for cost, latency, and token usage. It doesn't do much in the way of routing or key issuance, so it's often paired with another tool rather than used standalone.

What Self-Hosting Actually Costs You

Open source is free to download, not free to run. Before adopting any of these, you're signing up for:

• A deployment target (container, VM, or serverless function) that needs to stay up
• Config management for routing rules, rate limits, and provider keys
• Monitoring the gateway itself — if it goes down, every downstream app goes down with it
• Security review, since it now holds your provider credentials
• Upgrades as the project evolves, including breaking changes

None of this is prohibitive for a platform team with spare capacity. But for a product team whose actual goal is "give our app a stable Claude endpoint with per-key usage data," running and babysitting a gateway is a distraction from the product itself.

When a Managed Gateway Makes More Sense

If your use case is specifically Claude — not a multi-provider abstraction layer — a managed service can cover the same ground with less engineering overhead. SubToAPI turns your existing Claude access into a standard HTTPS API: you get application-scoped keys (sub_live_...), streaming, tool use, and usage metadata per key, with team seats managed in one dashboard.

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-3-5-sonnet-20241022",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Summarize this gateway comparison."}]
  }'

This covers the same ground as a self-hosted gateway — issuing distinct keys per app or environment, tracking usage, handling streaming — without you running infrastructure. See the quickstart, the messages endpoint docs, and streaming guide for implementation details. Plans start at €9/month on Solo, with Team (€19/seat) and Scale (€49/seat) tiers for multi-person setups, and a free trial at signup.

Picking the Right Approach

A rough decision guide:

• Multiple providers, heavy routing logic, in-house platform team → LiteLLM or Portkey, self-hosted
• Mainly observability, already have a gateway in place → Helicone as an add-on
• Primarily Claude, want per-app keys and usage data without ops work → a managed layer like SubToAPI
• Need tool use / function calling on top of Claude with clean key separation → check the tools documentation before building this yourself on raw provider keys

The honest trade-off is control versus maintenance. Open source gateways give you full control over routing logic and data residency, at the cost of running and securing another service. Managed gateways trade some of that control for speed and lower operational burden. Neither is universally "more correct" — it depends on whether gateway infrastructure is something your team wants to own.

Questions

Is an open source LLM API gateway free to run in production? The software is free, but you still pay for compute, monitoring, and the engineering time to configure, secure, and upgrade it — which is often the larger cost over time.

Do I need a gateway if I only use one model provider? If you only call one provider and don't need per-app key separation, usage tracking, or rate limiting, a gateway may be unnecessary overhead — a direct SDK call might be enough.

Can I switch from a self-hosted gateway to a managed one later? Yes. Since gateways typically sit behind a consistent API shape (like the Messages format), swapping the backend usually means changing a base URL and key, not rewriting application code.