What Is LLM Inference Cost? A Developer's Guide

LLM inference cost is the money you spend every time a trained model generates a response to a prompt. It's distinct from training cost, which is a one-time (or periodic) expense to build or fine-tune a model. Inference happens continuously, in production, every time a user sends a message, an app calls an API, or an agent runs a tool loop — which is why it's the number that actually shows up on your monthly bill.

For most teams building on top of hosted models like Claude, GPT, or Gemini, inference cost is calculated per token: a fixed price per million input tokens and a (usually higher) price per million output tokens. Your total cost is a function of how many requests you make, how long your prompts and responses are, and which model you use. Understanding this math is the first step to controlling it.

How LLM Inference Cost Is Calculated

Almost every commercial LLM provider prices inference the same way:

• Input tokens — everything you send: system prompt, conversation history, tool definitions, retrieved documents, user message.
• Output tokens — everything the model generates in response.
• Price per million tokens — usually quoted separately for input and output, since output generation is more computationally expensive.

A rough formula:

cost = (input_tokens / 1,000,000 × input_price)
     + (output_tokens / 1,000,000 × output_price)

If a model costs $3 per million input tokens and $15 per million output tokens, a request with 2,000 input tokens and 500 output tokens costs:

(2000 / 1,000,000 × 3) + (500 / 1,000,000 × 15)
= 0.006 + 0.0075
= $0.0135

That looks trivial per request, but multiply it by thousands of daily active users, each sending multiple messages, and it becomes a real line item. This is also why output tokens matter more than people expect — a model that "thinks out loud" or writes verbose responses can cost several times more than one prompted to be concise.

What Actually Drives Inference Cost

Model size and capability

Larger, more capable models cost more per token because they require more compute per forward pass. Choosing the biggest available model for every task is the single most common way teams overspend — a smaller model is often good enough for classification, extraction, or short-form generation.

Context length

Every token in your conversation history gets re-processed on each turn unless the provider offers prompt caching. Long system prompts, large tool schemas, and full chat histories sent on every call are a bigger cost driver than most people realize.

Output verbosity

Output tokens are typically priced 3–5x higher than input tokens. Prompting for shorter, more structured responses (JSON instead of prose, bullet points instead of paragraphs) has a direct and measurable effect on your bill.

Streaming vs. non-streaming

Streaming doesn't change the total token cost, but it changes perceived latency and lets you cut off generation early if you detect the answer is already complete — which does save tokens in practice.

Tool use and agent loops

Agents that call tools in a loop (search, then reason, then search again) multiply inference cost because each loop iteration re-sends the growing conversation history as input. A five-step agent loop can cost far more than a single completion, even if the final answer is short.

Retries and error handling

Failed requests, timeouts, and malformed outputs that trigger retries all consume tokens without producing value. Robust error handling isn't just about reliability — it's a cost control measure.

Inference Cost vs. Infrastructure Cost

If you're calling a hosted API, inference cost is your only line item — no GPUs to provision, no servers to keep warm. If you're self-hosting an open-weight model, inference cost also includes GPU rental or amortized hardware, plus engineering time to manage scaling, batching, and uptime. For most product teams, hosted APIs are cheaper in practice once you account for idle GPU time and the engineering overhead of running inference infrastructure — self-hosting only pays off at very high, consistent volume.

How to Estimate Your Own Inference Cost

Before shipping a feature, estimate cost with a simple back-of-envelope calculation:

1. Estimate average input tokens per request (include system prompt and history).
2. Estimate average output tokens per response.
3. Multiply by your expected daily request volume.
4. Apply the provider's per-million-token pricing.
5. Multiply by 30 for a monthly estimate.

Track actual usage once live — most providers, including SubToAPI, return token counts in the response metadata so you can log real cost per request rather than relying on estimates. See /docs/messages for the response shape and usage fields.

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 500,
    "messages": [{"role": "user", "content": "Summarize this in 3 bullet points."}]
  }'

The response includes usage.input_tokens and usage.output_tokens, which you can log per request to build an accurate cost dashboard instead of guessing.

Reducing Inference Cost Without Hurting Quality

• Trim system prompts and remove unused tool definitions from every call.
• Cap max_tokens to a realistic ceiling for the task.
• Summarize or truncate conversation history instead of resending it in full.
• Use a smaller model for simple, high-volume tasks and reserve larger models for complex reasoning.
• Cache repeated queries and static context where the provider supports it.

If you're piping requests through SubToAPI, /pricing shows the per-seat plans (Solo, Team, Scale) so you can separate your inference spend from your API management overhead, and /docs/quickstart walks through wiring up your first authenticated call.

questions

Is LLM inference cost the same as API cost? Yes, for hosted models. "Inference cost" is the general term for the compute cost of generating a response; "API cost" is what you pay a provider, which is their price for that same inference plus their margin.

Why is output more expensive than input in most pricing models? Generating tokens requires a sequential forward pass for each new token, while processing input tokens can be parallelized. That extra compute per output token is reflected in higher per-token pricing.

Does streaming reduce inference cost? Not directly — you're still billed for the same number of tokens. It can reduce cost indirectly if you stop generation early once you have enough output, since you avoid paying for unused tokens.