LLM Cost Benchmark: How to Compare Models Fairly

When people search for an "LLM cost benchmark," they usually want one of two things: a ready-made table comparing what GPT-4, Claude, Gemini, and open models cost per million tokens, or a way to run their own benchmark against their actual workload. The first is easy to find and almost useless, because published price sheets change monthly and say nothing about how many tokens your specific prompts and outputs actually consume. The second is what actually helps you make a decision, and it's what this article covers.

A real LLM cost benchmark isn't a spreadsheet of per-token prices pulled from vendor docs. It's a small experiment you run against your own prompts, with your own data, measuring what you'd actually pay in production — including the tokens you didn't think about, like system prompts, tool schemas, and retries.

Why list prices don't answer the question

Vendors publish price per million input tokens and per million output tokens. That's a real number, but it's not your cost. Two models with identical per-token pricing can produce very different bills because:

• Tokenization differs. The same English sentence can be 15% more or fewer tokens depending on the model's tokenizer. Non-English text and code often show bigger gaps.
• Verbosity differs. Some models write longer answers by default. If you're paying for output tokens, a chatty model can cost more even at a lower per-token rate.
• Retry and error rates differ. A model that fails your JSON schema validation 5% of the time means 5% of requests get re-sent, doubling cost on those calls.
• Context reuse differs. If your workload sends a large system prompt or tool definitions on every call, prompt caching support (or lack of it) changes the real cost dramatically.

A cost benchmark that ignores these factors will rank models in the wrong order for your use case.

What to actually measure

Build your benchmark around these four numbers, computed per task, not per token:

1. Cost per completed task. Run the same set of representative prompts through each model and record total tokens (input + output) times the model's published rate, including retries.
2. Failure-adjusted cost. If a response needs a retry, a follow-up call, or human correction, add that cost back in. A cheaper model with a higher failure rate is often more expensive in practice.
3. Latency-weighted cost. If your product needs sub-2-second responses and a cheap model takes 6 seconds, you may need to fall back to a faster (pricier) model for some fraction of traffic. Blend the cost accordingly.
4. Cost variance. Look at the spread, not just the average. A model that's usually cheap but occasionally generates a very long response can blow your budget on outliers — check p95, not just mean.

A simple benchmark script

Here's a minimal pattern for benchmarking cost per task across prompts, generic enough to adapt to any provider's SDK:

const prompts = loadYourTestPrompts(); // 50-200 real examples
const results = [];

for (const prompt of prompts) {
  const start = Date.now();
  const response = await callModel(prompt); // your provider call
  const latencyMs = Date.now() - start;

  const cost =
    (response.usage.input_tokens / 1_000_000) * INPUT_RATE +
    (response.usage.output_tokens / 1_000_000) * OUTPUT_RATE;

  results.push({
    prompt: prompt.id,
    inputTokens: response.usage.input_tokens,
    outputTokens: response.usage.output_tokens,
    cost,
    latencyMs,
    passedValidation: validateOutput(response),
  });
}

summarize(results); // mean cost, p95 cost, failure rate, mean latency

Run this against every model you're evaluating with the exact same prompt set. The output isn't "Model A costs $X per million tokens" — it's "Model A costs €0.014 per completed, validated task at p95 latency of 1.8s." That's the number you can actually put in a budget.

Building a repeatable benchmark, not a one-off test

A single run tells you almost nothing, because LLM outputs vary between calls. Run each prompt 3-5 times per model and average. Also re-run the whole benchmark monthly — providers change default behavior, deprecate models, and adjust pricing without much warning, so a benchmark from six months ago is not a reliable guide today.

Keep your test prompt set representative of production, not a generic benchmark suite. If your app does customer support summarization, benchmark with real (anonymized) support tickets, not trivia questions. Cost per token is a commodity number; cost per your actual task is the one that matters.

Where usage metadata gets complicated

One practical obstacle to running this kind of benchmark: getting clean, per-request usage data. If you're comparing models across different provider consoles, you often end up stitching together token counts from different dashboards with different reporting delays, which makes it hard to compute cost per task accurately across a team.

This is one of the reasons we built SubToAPI — it turns your existing Claude access into an HTTPS API with per-key usage metadata on every response, so you can log input tokens, output tokens, and cost per call directly in your own benchmark script instead of reconciling numbers across dashboards. If you're benchmarking Claude specifically as part of a wider comparison, the quickstart shows the request shape, and pricing has the plan details (Solo €9, Team €19/seat, Scale €49/seat, free trial included).

Putting it together

A trustworthy LLM cost benchmark has three properties: it uses your own prompts, it accounts for failures and retries, and it's re-run often enough to stay current. Skip the exercise of comparing sticker prices across providers — build the 50-line script above, point it at the models you're actually considering, and let the numbers from your own workload make the decision.

Questions

Is cost per token a good enough metric on its own? No. It ignores tokenizer differences, output verbosity, and failure rates, all of which can shift the real cost per task by 30% or more between models with identical list prices.

How many test prompts do I need for a reliable benchmark? 50-200 representative prompts from your actual workload, run 3-5 times each, is usually enough to see a stable difference between models and catch high-variance outliers.

How often should I re-run an LLM cost benchmark? Monthly, or immediately after a provider announces a pricing change or new model version — both happen frequently enough that a six-month-old benchmark is unreliable.