How to Build a Document QA System with Claude

What a Document QA System Actually Needs

A document QA system lets users ask natural-language questions about a set of files — contracts, manuals, reports, support tickets — and get accurate answers grounded in that content, not hallucinated from general knowledge. To build one with Claude, you need four pieces: a way to extract and chunk document text, a retrieval step that finds the relevant chunks for a given question, a prompt that forces Claude to answer only from the provided context, and an API layer that handles the actual calls.

This guide walks through each piece with working code, then covers the common failure modes (wrong answers from irrelevant chunks, context window overflow, missing citations) and how to fix them. By the end you'll have a working pipeline you can adapt to any document type.

Step 1: Extract and Chunk Your Documents

Claude doesn't read PDFs or Word files directly in a QA pipeline — you extract plain text first, then split it into chunks small enough to retrieve individually but large enough to preserve context.

A simple, effective chunking strategy for most business documents:

function chunkText(text, maxChars = 1500, overlap = 200) {
  const chunks = [];
  let start = 0;
  while (start < text.length) {
    const end = Math.min(start + maxChars, text.length);
    chunks.push(text.slice(start, end));
    start += maxChars - overlap;
  }
  return chunks;
}

The overlap matters — without it, a sentence that spans two chunks can get cut in a way that makes neither chunk useful for answering a question about it. For structured documents (contracts, manuals), chunk by section or heading instead of raw character count when possible — it produces more coherent retrieval results.

Step 2: Build Retrieval

You don't need a full vector database to get started. For small-to-medium document sets (a few hundred pages), a straightforward approach works fine:

1. Embed each chunk using any embedding model.
2. Embed the incoming question.
3. Compute cosine similarity and take the top 3–5 chunks.

function cosineSimilarity(a, b) {
  const dot = a.reduce((sum, v, i) => sum + v * b[i], 0);
  const magA = Math.sqrt(a.reduce((sum, v) => sum + v * v, 0));
  const magB = Math.sqrt(b.reduce((sum, v) => sum + v * v, 0));
  return dot / (magA * magB);
}

function topChunks(questionEmbedding, chunkEmbeddings, chunks, k = 4) {
  const scored = chunks.map((chunk, i) => ({
    chunk,
    score: cosineSimilarity(questionEmbedding, chunkEmbeddings[i]),
  }));
  return scored.sort((a, b) => b.score - a.score).slice(0, k).map((s) => s.chunk);
}

If you're working with a handful of documents and don't want to manage embeddings at all, you can skip retrieval entirely and pass whole documents into Claude's context window directly — more on that trade-off below.

Step 3: Write a Prompt That Forces Grounded Answers

This is the part people get wrong most often. If you just paste chunks into the prompt and ask a question, Claude will sometimes fill gaps with general knowledge instead of saying "not in the document." You need explicit instructions and a structure that makes citation easy.

You are a document QA assistant. Answer the user's question using ONLY the
context below. If the answer isn't in the context, say "I don't see that
information in the provided documents."

For every claim in your answer, cite the source chunk number in brackets, e.g. [2].

Context:
[1] <chunk text>
[2] <chunk text>
[3] <chunk text>

Question: {user_question}

Numbering chunks and requiring bracket citations gives you two benefits: it reduces hallucination (the model is less likely to assert something it can't point to), and it gives your UI a way to show "source: chunk 2" links back to the original document, which builds user trust.

Step 4: Call the API

Once you have your context block assembled, the actual API call is a single request. Here's what it looks like against SubToAPI, which turns your existing Claude access into a standard HTTPS API with API keys, so you don't need separate billing or infrastructure to get this running:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "max_tokens": 1024,
    "messages": [
      {
        "role": "user",
        "content": "You are a document QA assistant...\n\nContext:\n[1] ...\n[2] ...\n\nQuestion: What is the notice period for termination?"
      }
    ]
  }'

For production use, you'll want streaming so the answer renders as it generates rather than after a multi-second wait — see /docs/streaming for the implementation. If your QA system also needs to trigger actions (like fetching a specific document section on demand instead of pre-chunking everything), Claude's tool use support is worth exploring too — covered in /docs/tools.

Handling Context Window Limits

If your document set is small enough to fit entirely in Claude's context window, skip retrieval and just paste the whole thing in — it's simpler and avoids retrieval errors entirely. This works well for single contracts, single manuals, or small knowledge bases under roughly 100 pages.

For larger sets, retrieval isn't optional, but you can improve accuracy by:

• Re-ranking retrieved chunks with a cheap second pass before sending the top ones to Claude.
• Expanding chunks to include a paragraph before/after the matched section, so partial matches still have surrounding context.
• Logging low-confidence retrievals (low similarity scores) and flagging those answers as uncertain in your UI.

Getting From Prototype to Production

A document QA prototype is a weekend project. Making it reliable for real users means handling retries, rate limits, usage tracking per customer (if you're billing for it), and API key management for your team. SubToAPI handles the API layer — issuing scoped sub_live_... keys, streaming, and usage metadata — so you can focus on chunking, retrieval quality, and prompt tuning instead of plumbing. Check /docs/quickstart to get a key running in minutes, or /pricing if you're evaluating for a team.

Questions

Do I need a vector database to build this? No — for small document sets, in-memory cosine similarity over embeddings works fine. Add a vector database only once you're dealing with thousands of chunks and need fast lookups at scale.

How do I stop Claude from making up answers not in the documents? Use an explicit system instruction restricting answers to the provided context, require bracket citations tied to chunk numbers, and have the model explicitly say when information isn't present.

Can I just paste the whole document instead of chunking and retrieving? Yes, if it fits in the context window — this is simpler and more accurate for single documents or small sets. Chunking and retrieval become necessary once your total document volume exceeds what fits in one request.