How to Chunk Long Documents for Claude API
When a document is too large to fit in a single Claude API request — or too large to process cost-effectively in one call — you need to split it into chunks that preserve meaning, fit within context limits, and still produce coherent output when reassembled. The right approach depends on document type (prose, code, tables, transcripts) and what you're doing with the output (summarization, extraction, Q&A, translation).
The short answer: don't chunk by raw character count alone. Chunk by semantic boundaries (paragraphs, sections, headings) with a target token budget per chunk, add a small overlap between chunks for context continuity, and carry forward a compact "memory" of prior chunks when the task requires cross-chunk coherence (like summarizing a 200-page report). Below is a practical breakdown of how to do this correctly.
Why Chunking Matters Even With Large Context Windows
Claude's models support large context windows, but that doesn't mean you should always send the whole document in one shot:
- Cost: Input tokens are billed per request. A 150,000-token document sent repeatedly (e.g., in a multi-turn chat) gets expensive fast.
- Latency: Larger inputs mean longer time-to-first-token, especially with streaming.
- Accuracy: Models can lose focus on details buried in the middle of very long inputs ("lost in the middle" effect), even within supported context limits.
- Rate limits: Some plans or API tiers cap tokens per minute, not just per request.
Chunking isn't just a workaround for context limits — it's often the better engineering choice even when the whole document would technically fit.
Step 1: Pick a Chunking Strategy Based on Document Structure
Fixed-size chunking (e.g., every 2,000 tokens) is the simplest but worst option for quality — it cuts sentences and sections mid-thought.
Structure-aware chunking respects natural boundaries:
- Markdown/HTML: split on headings (
##,<h2>), then paragraphs within sections. - Legal/technical docs: split on numbered clauses or sections.
- Transcripts: split on speaker turns or timestamps.
- Code: split on function/class boundaries, never mid-function.
- CSV/tabular: chunk by row ranges, always keeping the header row with each chunk.
Recursive chunking is a good default: try to split on the largest structural unit (section), and if that's still too big, recursively split on the next level down (paragraph, then sentence).
function chunkText(text, maxTokens = 1500) {
const sections = text.split(/\n(?=#{1,3} )/); // split on markdown headings
const chunks = [];
for (const section of sections) {
if (estimateTokens(section) <= maxTokens) {
chunks.push(section);
} else {
// recurse into paragraphs
const paragraphs = section.split('\n\n');
let buffer = '';
for (const p of paragraphs) {
if (estimateTokens(buffer + p) > maxTokens) {
chunks.push(buffer);
buffer = p;
} else {
buffer += '\n\n' + p;
}
}
if (buffer) chunks.push(buffer);
}
}
return chunks;
}
function estimateTokens(str) {
return Math.ceil(str.length / 4); // rough heuristic, ~4 chars per token
}This heuristic is approximate — for production use, run a real tokenizer count before finalizing chunk sizes, since underestimating can cause requests to exceed limits.
Step 2: Add Overlap Between Chunks
If a sentence or idea spans a chunk boundary, the model processing chunk N+1 has no idea what came right before it. A 10–15% overlap (by token count) between consecutive chunks fixes most of this:
function addOverlap(chunks, overlapTokens = 150) {
return chunks.map((chunk, i) => {
if (i === 0) return chunk;
const prevTail = chunks[i - 1].slice(-overlapTokens * 4); // ~4 chars/token
return prevTail + '\n\n' + chunk;
});
}Overlap matters most for summarization and extraction tasks. It matters less for strict Q&A over isolated sections where bleed-through context could actually confuse the answer.
Step 3: Decide How to Reassemble Output
Chunking the input is half the problem — you also need a strategy for combining outputs:
- Map-reduce summarization: summarize each chunk independently, then summarize the summaries. Works well for very long documents but can lose fine detail.
- Rolling context: pass a short summary of everything processed so far along with the next chunk, so the model builds a running understanding. Better for narrative coherence, costs a bit more per call.
- Independent extraction: for tasks like pulling structured data (dates, names, line items) from each chunk, process chunks independently and merge the structured results — no cross-chunk context needed.
Pick based on the task. Don't use rolling context for simple extraction — it adds cost and latency with no quality benefit.
Step 4: Build the Pipeline Around Claude API Calls
Once chunks are ready, each one becomes a request. If you're calling Claude through SubToAPI, the pattern looks like this:
curl https://api.subtoapi.app/v1/messages \
-H "Authorization: Bearer $SUBTOAPI_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet-4",
"max_tokens": 1024,
"messages": [
{"role": "user", "content": "Summarize this section:\n\n" }
]
}'For map-reduce pipelines, run chunk requests in parallel to cut total processing time, then make a final call with the combined summaries. SubToAPI gives each application its own sub_live_... key with usage metadata per request, which makes it straightforward to track token consumption across a chunked batch job and see exactly where your cost is going — useful when you're debugging whether your chunk size is too large or your overlap is too generous. See /docs/messages for request formatting and /docs/streaming if you're streaming per-chunk responses back to a UI as they complete.
Practical Chunk Size Guidelines
- Summarization: 1,500–3,000 tokens per chunk with 10–15% overlap.
- Extraction (structured data): as large as fits comfortably — fewer chunks means less merge logic, overlap not needed.
- Q&A over a knowledge base: smaller chunks (300–800 tokens) paired with a retrieval step, since you want precise, isolated passages, not broad sections.
- Translation: chunk by paragraph, never mid-sentence, minimal overlap needed since translation doesn't require cross-chunk memory.
Questions
Does Claude have a hard document size limit I need to chunk around? Yes — every model has a maximum context window measured in tokens, and that limit includes both your input and the expected output. Check the current limits for your model before assuming a document fits in one call.
Should I always use overlap between chunks? No. Overlap helps when meaning spans chunk boundaries (summarization, narrative text), but it's unnecessary and wasteful for independent extraction tasks where each chunk is self-contained, like pulling line items from separate invoice pages.
How do I know if my chunks are too large? If output quality drops on long chunks (details from the middle get skipped or summaries feel shallow), reduce chunk size. Test with a fixed document and compare output quality at 1,000, 2,000, and 4,000-token chunk sizes to find the sweet spot for your use case.