How to Design an API Gateway: Key Decisions to Get Right

Designing an API gateway means deciding, before you write any code, what problems it needs to solve, where it sits in your traffic flow, and which responsibilities it owns versus which it delegates to your backend services. Get this wrong and you end up with a gateway that's either too thin to be useful or so bloated it becomes the single point of failure and the bottleneck for every feature ship.

This article walks through the concrete design decisions in order: traffic patterns, routing strategy, authentication model, rate limiting, request/response transformation, and observability. Each section includes the tradeoffs so you can make an informed call for your specific system, rather than copying a reference architecture that doesn't match your constraints.

Start With Traffic Patterns, Not Technology

Before picking a gateway product or framework, map out what actually flows through it:

• North-south traffic: external clients (web, mobile, third-party integrations) hitting your services.
• East-west traffic: internal service-to-service calls. Most designs should not route this through the same gateway — it adds latency and a shared failure point for internal calls that don't need public-facing concerns like OAuth or CORS.
• Streaming vs request/response: if you're proxying long-lived streaming responses (Server-Sent Events, WebSockets, chunked HTTP), your gateway needs to support backpressure and not buffer the entire response before forwarding it.

The design mistake teams make most often is building one gateway to handle everything: public API, internal mesh, and admin tooling. Separate these concerns early, even if they share infrastructure.

Decide Your Routing Strategy

Routing is the core job of a gateway: mapping an incoming request to a backend service. Two common approaches:

Path-based routing — /users/ goes to the users service, /orders/ goes to the orders service. Simple, predictable, easy to debug.

Version/header-based routing — routing by Accept header, API version, or client type. More flexible but harder to trace; you need good logging to know which backend actually served a request.

A practical rule: keep routing rules declarative (config, not code) so you can add or move a service without redeploying the gateway itself. Most teams regret hardcoding route logic in application code inside the gateway process.

Choose the Authentication Model Upfront

Authentication design at the gateway layer typically falls into one of three patterns:

1. API keys — simplest, good for service-to-service or B2B integrations where you control both ends.
2. OAuth2 / JWT validation — the gateway validates a token's signature and claims, then forwards a trusted identity header downstream. Backends never see raw tokens.
3. mTLS — for high-security internal or partner traffic where certificate-based identity is required.

The design decision that matters most: decide once where authentication happens and don't duplicate it. If the gateway validates the token, backend services should trust the forwarded identity and not re-validate independently — that just adds latency without adding security, and creates two places that can disagree about who's authenticated.

If you're exposing a third-party model or service as your own API — for example, giving your application a stable API key instead of routing raw provider credentials through your codebase — this is exactly the pattern a product like SubToAPI handles: it sits in front of your Claude access and issues scoped sub_live_... keys per application, so your gateway design doesn't need to reinvent key issuance and rotation for that integration. See the quickstart for how that looks in practice.

Rate Limiting: Pick the Right Granularity

Rate limiting at the gateway protects backends from overload and enforces plan limits. The design questions to answer:

• Per what? Per API key, per IP, per endpoint, or a combination. Per-key is usually right for authenticated APIs; per-IP is a fallback for anonymous traffic.
• Fixed window or sliding window? Fixed windows are cheaper to compute but allow bursts at window boundaries. Sliding windows or token buckets are smoother but cost more state.
• Where does state live? In-memory rate limiting doesn't work across multiple gateway instances — you need a shared store (Redis is the common choice) or you'll get inconsistent limits under load.

Return clear rate-limit headers (X-RateLimit-Remaining, Retry-After) so clients can back off intelligently instead of hammering a 429 endpoint.

Decide What Transformation Belongs at the Gateway

Gateways can rewrite requests and responses — stripping headers, reshaping payloads, aggregating multiple backend calls into one response. The design tradeoff:

• Light transformation (auth header injection, CORS headers, response compression) belongs at the gateway — it's cross-cutting and stateless.
• Business-logic transformation (combining data from three services into one response shape) usually belongs in a dedicated backend-for-frontend layer, not the gateway itself. Mixing business logic into gateway config makes it hard to test and easy to break silently.

Build Observability In From Day One

A gateway sees every request, which makes it the best place to capture:

• Latency per route and per backend
• Error rates broken out by status code
• Usage per API key (critical if you bill by usage or seat)

Design this as structured logs and metrics from the start rather than bolting it on later — retrofitting observability into a gateway that's already routing production traffic is far riskier than building it from the first deployment. If your gateway also functions as a billing boundary, usage metadata per key becomes a first-class design requirement, not an afterthought — this is one of the reasons SubToAPI ships usage metadata and streaming support alongside routing, so teams don't have to build that instrumentation themselves for every integration.

A Minimal Reference Flow

# Client calls your gateway with an application-scoped key
curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "max_tokens": 512,
    "messages": [{"role": "user", "content": "Summarize this ticket."}]
  }'

The gateway validates the key, applies rate limits, records usage, and forwards the request — the same pattern applies whether you're building this in-house or using a hosted layer. If you're evaluating whether to build or adopt one for exposing Claude access as an API, the pricing page outlines what's included at each tier.

Checklist Before You Build

• Separate north-south and east-west traffic
• Keep routing rules declarative and out of application code
• Authenticate once, trust the forwarded identity downstream
• Rate-limit per key with a shared state store across instances
• Keep business logic out of gateway transformation rules
• Instrument latency, errors, and per-key usage from day one

questions

Should I build my own API gateway or use a managed product? Build your own if routing and auth logic are simple and stable; use a managed product when you need TLS termination, rate limiting, and observability without maintaining that infrastructure yourself, especially as traffic and team size grow.

How many gateways should a system have? Usually one for external (north-south) traffic and, if needed, a separate lightweight proxy or service mesh for internal (east-west) traffic — combining both into one layer adds unnecessary coupling and risk.

What's the biggest mistake in gateway design? Putting business logic or backend-specific transformation into the gateway layer. It should stay generic and stateless — authentication, routing, and rate limiting — while business rules live in the services behind it.