LLM API Gateway: Open Source, Self-Hosted, or Hosted?

What an LLM API Gateway Actually Does

An LLM API gateway sits between your application and one or more model providers. It gives you a single, stable endpoint that handles authentication, request routing, retries, rate limiting, logging, and often a unified request/response format across providers like OpenAI, Anthropic, and open-weight models served locally. If you're searching for "open source self-hosted," you almost certainly want one (or more) of these: full control over your data, no per-request markup from a third party, the ability to run the gateway inside your own VPC, and freedom to swap providers without rewriting your app.

The short answer: yes, solid open source self-hosted LLM gateways exist — LiteLLM Proxy, Portkey's open source gateway, and Kong's AI Gateway plugins are the most mature. They're free to run, but "free" means you pay in ops time: deployment, scaling, patching, and uptime become your problem. Whether that trade is worth it depends on your team size, compliance requirements, and how much engineering time you want to spend on infrastructure instead of product.

Open Source Gateway Options Worth Evaluating

LiteLLM Proxy

LiteLLM is the most widely deployed option for teams that want an OpenAI-compatible interface in front of 100+ model providers. It runs as a lightweight Python service, supports virtual API keys, budget limits per key, fallback routing between models, and request logging to tools like Langfuse or Datadog. A minimal self-hosted config looks like:

model_list:
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
general_settings:
  master_key: sk-local-master-key

Run it with litellm --config config.yaml and you get an OpenAI-shaped /chat/completions endpoint backed by whichever provider you configured.

Portkey Gateway

Portkey's open source gateway focuses on reliability features: automatic retries, load balancing across API keys, semantic caching, and guardrails for structured output. It's a good fit if your main pain point is flaky upstream providers rather than multi-model routing.

Kong AI Gateway

If you already run Kong for regular API traffic, its AI Gateway plugins let you add LLM routing, token-based rate limiting, and prompt transformation on top of infrastructure you're already operating. The advantage is consistency — one gateway stack for both your REST APIs and your LLM traffic.

Helicone (proxy mode)

Helicone is primarily an observability layer, but it can run as a self-hosted proxy in front of any provider, giving you request logs, cost tracking, and latency metrics without changing your API calls beyond the base URL.

All four are genuinely open source and genuinely self-hostable. None of them, on their own, give you a polished multi-tenant dashboard, seat-based billing, or a packaged onboarding flow — you build that layer yourself if you need it.

What Self-Hosting Actually Costs You

Running your own gateway isn't just docker run. Over time you end up owning:

• Uptime — if the gateway container crashes, every app that depends on it goes down simultaneously
• Scaling — concurrent request limits on the gateway host itself, separate from provider-side limits
• Security patching — CVEs in the gateway's dependencies, TLS certificate renewal, key rotation
• Observability — you still need to wire up logging, alerting, and usage dashboards yourself
• Team access control — most open source gateways give you API keys, not full role-based team management out of the box

For a solo project or an internal tool with one or two engineers, this overhead is manageable. For a product with paying customers, a growing team, and on-call expectations, it becomes a second piece of infrastructure to maintain alongside your actual application.

When a Hosted Gateway Makes More Sense

If your actual goal is narrower — you specifically want to turn Claude access into a clean, reliable HTTPS API with application-level keys, streaming, tool use, and per-key usage metadata, without running and patching a gateway yourself — a managed option like SubToAPI covers that specific case. It's not an open source self-hosted gateway; it's a hosted SaaS that issues sub_live_... keys scoped to your account, so you skip the infra work entirely:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-3-5-sonnet-20241022",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Summarize this ticket"}]
  }'

Streaming, tool use, and team seats are documented in /docs/streaming, /docs/tools, and /docs/messages, with a full walkthrough at /docs/quickstart. Plans start at Solo €9, with Team (€19/seat) and Scale (€49/seat) tiers for shared usage dashboards — see /pricing. There's a free trial at /signup if you want to compare the setup time against running LiteLLM yourself.

The honest framing: choose self-hosted open source if you need multi-provider routing, full infra control, or have compliance reasons to keep everything in your own cloud. Choose a hosted option if your scope is "give my app a reliable API key for Claude" and you'd rather not operate another service.

Decision Checklist

• Do you need to route across multiple model providers, not just one? → Lean open source (LiteLLM, Portkey)
• Do you need the gateway running inside your own VPC for compliance? → Self-host
• Is your main need per-key usage tracking and team seats for Claude specifically, with no ops overhead? → A managed API layer fits better
• Do you have spare on-call capacity for another production service? → If not, factor that into the "free" cost of open source

Questions

Is a self-hosted LLM gateway really free? The software is free, but you pay in compute, maintenance time, and on-call risk — budget for that before assuming it's cheaper than a managed option.

Can I run an open source gateway in front of Claude specifically? Yes — LiteLLM, Portkey, and Kong all support Anthropic as a provider; you configure your Anthropic API key as an environment variable in the gateway config.

What's the simplest alternative if I don't want to manage a gateway at all? A hosted API layer purpose-built for Claude, like SubToAPI, gives you scoped keys, streaming, and usage metadata without deploying infrastructure — see /docs/quickstart to compare setup time.