A Python Tool for Claude API Token Counting

If you're building anything non-trivial with Claude, you'll eventually hit two problems: requests that silently get truncated because you exceeded the context window, and a monthly bill that's hard to explain because you don't know how many tokens your prompts actually consume. The fix for both is the same — count tokens before you send the request, not after.

This article walks through building a small Python tool that counts tokens for Claude API calls, covers the official counting endpoint, and explains where a pre-flight estimate stops being accurate (and what to do instead when you need exact numbers).

Why token counting matters

Claude models have a fixed context window, and every message, system prompt, and tool definition eats into it. If you're building RAG pipelines, chat history management, or batch summarization jobs, you need to know roughly how many tokens a payload will use so you can:

• Decide whether to trim conversation history before hitting the limit
• Estimate cost per request before committing to a batch job
• Choose between models based on expected input/output size
• Debug "context length exceeded" errors before they happen in production

Token counts aren't the same as word counts or character counts. Claude's tokenizer splits text into subword units, and punctuation, whitespace, and non-English text all tokenize differently than plain English prose. That's exactly why a dedicated counting tool is worth having instead of guessing with len(text.split()).

Option 1: Use Anthropic's official token counting endpoint

Anthropic exposes a count_tokens call via their Python SDK that returns an exact count for a given set of messages, without actually generating a completion. This is the most reliable approach because it uses the same tokenizer the model will use at inference time.

import anthropic

client = anthropic.Anthropic(api_key="YOUR_API_KEY")

response = client.messages.count_tokens(
    model="claude-sonnet-4-20250514",
    messages=[
        {"role": "user", "content": "Summarize this document in three sentences."}
    ],
    system="You are a precise technical summarizer."
)

print(response.input_tokens)

This costs nothing to call and doesn't consume your generation quota, so it's safe to run on every request in a pre-flight check. Wrap it in a small helper:

def fits_in_budget(messages, system, max_tokens=180000):
    result = client.messages.count_tokens(
        model="claude-sonnet-4-20250514",
        messages=messages,
        system=system,
    )
    return result.input_tokens <= max_tokens, result.input_tokens

Call this before assembling your final request, and you can decide to truncate history, drop attachments, or split the job into multiple calls.

Option 2: Build a lightweight estimator for offline use

Sometimes you don't want a network round trip just to estimate size — for example, when batching thousands of documents and you only need a rough cutoff. A common approximation for English text is roughly 4 characters per token, which is close enough for filtering decisions (not billing decisions):

def estimate_tokens(text: str) -> int:
    return max(1, len(text) // 4)

def estimate_message_tokens(messages: list[dict]) -> int:
    total = 0
    for msg in messages:
        content = msg.get("content", "")
        if isinstance(content, str):
            total += estimate_tokens(content)
        elif isinstance(content, list):
            for block in content:
                if block.get("type") == "text":
                    total += estimate_tokens(block.get("text", ""))
    return total

Use this as a cheap first-pass filter — "is this document obviously too long?" — and only call the real count_tokens endpoint for payloads that pass the rough check. This two-tier approach keeps batch jobs fast without sacrificing accuracy where it matters.

Option 3: Wrap it into a CLI tool

If your team repeatedly needs to check token counts for prompts stored in files, a small CLI saves everyone from re-writing the same snippet:

import sys
import anthropic

client = anthropic.Anthropic()

def main():
    path = sys.argv[1]
    with open(path, "r") as f:
        text = f.read()

    result = client.messages.count_tokens(
        model="claude-sonnet-4-20250514",
        messages=[{"role": "user", "content": text}],
    )
    print(f"{path}: {result.input_tokens} tokens")

if __name__ == "__main__":
    main()

Run it as python count.py prompt.txt in CI to flag prompts that grow past a threshold before they ship.

Where pre-flight counting isn't enough

Counting input tokens tells you about the request side, but it says nothing about output tokens, which you only know after generation completes — and output tokens are usually billed at a different rate than input. If you're trying to track actual spend rather than just avoiding context overflows, you need usage data returned with every response, not an estimate computed beforehand.

This is where routing requests through SubToAPI helps if you're already on a Claude plan and want API access without separately provisioning API credits. Every response includes usage metadata (input and output token counts) alongside the completion, so you get exact per-request numbers without writing your own counting logic for billing purposes:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Summarize this report."}]
  }'

The response includes a usage object with exact counts, which you can log per application key in your own dashboard or pull from SubToAPI's dashboard directly — useful when multiple team members or services share the same underlying Claude access. See /docs/messages for the full response shape and /docs/streaming if you're counting tokens on streamed responses, where usage is reported at the end of the stream rather than upfront.

So the practical pattern is: use Python-side counting (official or estimated) to stay under context limits before sending a request, and rely on the API's returned usage data for actual cost tracking afterward. Don't try to compute cost from an input-side estimate alone — it will drift from reality once output tokens and caching are factored in.

Putting it together

A reasonable setup for most teams:

1. Rough character-based estimate for fast filtering in batch pipelines
2. Exact count_tokens call before any request close to the context limit
3. Usage metadata from the actual API response for cost tracking and reporting

This three-layer approach avoids both failure modes — overflowing the context window and flying blind on cost — without adding meaningful latency to normal requests.

questions

Does counting tokens with the Python SDK cost anything? No. The count_tokens call doesn't generate a completion, so it doesn't consume generation quota or incur generation cost — it only returns the tokenized count for the input you provide.

Is the 4-characters-per-token estimate accurate enough for billing? No. It's a rough filter for batch processing, not a billing tool. Non-English text, code, and heavy punctuation tokenize differently, so use the official counting endpoint or actual response usage data for anything cost-related.

Can I count tokens for streamed responses the same way? Input tokens can be counted before streaming starts with count_tokens. Output token counts for a stream are only available once the stream completes, reported in the final usage data — see /docs/streaming for the exact event shape.