How to Build a Document QA App with Claude

Building a document QA app with Claude means combining three pieces: a way to get relevant text out of your documents, a prompt that forces Claude to answer only from that text, and an API layer that handles the request/response cycle reliably. This guide walks through each piece with working code, from raw PDFs to a streaming answer with citations.

The core architecture is simple: extract and chunk your documents, retrieve the chunks relevant to a user's question, pass them to Claude as context, and ask it to answer strictly from that context. This is retrieval-augmented generation (RAG), and Claude is well suited to it because of its long context window and strong instruction-following — you can often skip a vector database entirely for small document sets and just stuff the whole document into the prompt.

Step 1: Decide if you need retrieval at all

If your documents fit comfortably under ~150k tokens total, don't build a retrieval pipeline. Just paste the full document (or set of documents) into the prompt and ask Claude the question directly. This is faster to build, easier to debug, and avoids chunking artifacts where the answer spans a boundary you split on.

Use retrieval only when:

• Your document set exceeds the model's practical context budget
• You have many documents and need to search across them, not just one file
• You need low latency and can't afford to send megabytes of text per request

Step 2: Extract and chunk text

For PDFs, use pdf-parse or pdfjs-dist in Node, or pdfplumber in Python. Strip headers/footers and page numbers before chunking — they add noise without adding signal.

Chunk by semantic boundaries (paragraphs, sections) rather than fixed character counts when possible. A reasonable default:

function chunkText(text, maxTokens = 800, overlap = 100) {
  const paragraphs = text.split(/\n\s*\n/);
  const chunks = [];
  let current = "";

  for (const para of paragraphs) {
    if ((current + para).length / 4 > maxTokens) {
      chunks.push(current.trim());
      current = current.slice(-overlap * 4) + para;
    } else {
      current += "\n\n" + para;
    }
  }
  if (current.trim()) chunks.push(current.trim());
  return chunks;
}

The / 4 is a rough tokens-per-character estimate for English text — good enough for sizing, not for billing.

Step 3: Retrieve relevant chunks

For retrieval, embed each chunk once at ingestion time and store vectors in something simple like a Postgres pgvector column, SQLite with a vector extension, or an in-memory array if your corpus is small (under a few thousand chunks, cosine similarity in plain JS is fast enough).

function cosineSimilarity(a, b) {
  let dot = 0, normA = 0, normB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}

function topChunks(queryVector, chunkVectors, k = 5) {
  return chunkVectors
    .map((c) => ({ ...c, score: cosineSimilarity(queryVector, c.vector) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k);
}

Pick an embedding model from whichever provider you're already using for embeddings — Claude doesn't currently expose a native embeddings endpoint, so this part of the pipeline sits outside the Claude API itself.

Step 4: Write a grounded prompt

The prompt is where most document QA apps succeed or fail. Be explicit about three things: use only the provided context, cite the source, and say "I don't know" when the context doesn't contain the answer.

const systemPrompt = `You are a document assistant. Answer the user's question using ONLY the context below.
If the answer is not in the context, say "I couldn't find that in the provided documents."
Always cite which source chunk you used, like [Source 2].

Context:
${retrievedChunks.map((c, i) => `[Source ${i + 1}] (${c.filename})\n${c.text}`).join("\n\n")}`;

This single instruction set — "answer only from context, cite the source, say when you don't know" — eliminates most hallucination problems without any fine-tuning.

Step 5: Call the API and stream the answer

Once you have context assembled, the request itself is a standard messages call. If you're running this through SubToAPI, the call looks like this:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "max_tokens": 1024,
    "system": "You are a document assistant...",
    "messages": [
      {"role": "user", "content": "What termination clause applies to late payment?"}
    ]
  }'
}'

For a chat-style UI, stream the response so users see the answer appear token by token instead of waiting for the full reply — this matters a lot for documents where answers can run several paragraphs. See /docs/streaming for the full event format if you're wiring this into a frontend.

SubToAPI gives you an application API key (sub_live_...) that you can embed in your backend without managing Claude credentials directly, plus usage metadata per request — useful when you're billing document QA as a feature inside a larger product. Full request/response shapes are in /docs/messages, and the five-minute setup is at /docs/quickstart.

Step 6: Handle multi-document questions

When a question might span multiple documents (e.g., "compare the indemnification clauses across these three contracts"), retrieve top chunks from each document separately rather than pooling and re-ranking everything together — this prevents one long document from drowning out a shorter one in the similarity rankings. Label each chunk clearly with its source filename in the prompt so Claude can distinguish them in its answer.

If your QA app needs to look things up dynamically — checking a live database, fetching a related record, or running a calculation — define that as a tool rather than stuffing more text into the prompt. See /docs/tools for how tool calling works end to end.

Step 7: Add answer verification

For production document QA, add a lightweight second pass: ask Claude to verify its own answer against the cited source before returning it to the user. This catches cases where the model cites a chunk but actually pulled the answer from general knowledge instead. It costs one extra call but meaningfully reduces subtle inaccuracies in compliance-sensitive use cases like contract or policy review.

Pulling it together

A production-grade document QA app is really just: ingest → chunk → embed → retrieve → prompt → stream → verify. Each step is independently simple; the complexity comes from tuning chunk size, retrieval depth, and prompt wording for your specific document types. Start with the whole-document-in-context approach, measure where it breaks down, and only add retrieval once you have evidence you need it.

questions

Do I need a vector database to build a document QA app with Claude? Not always. If your documents are small enough to fit in the context window, skip retrieval entirely and pass the full text. Add a vector store only once document volume forces it.

How do I stop Claude from making up answers not in the document? Use an explicit system prompt that instructs Claude to answer only from provided context, cite the source chunk, and say when the answer isn't present. This single change removes most hallucinations.

Can I stream document QA answers to a frontend in real time? Yes — the messages API supports streaming responses token by token, which is recommended for longer document answers so users don't wait on a blank screen. See /docs/streaming for the implementation details.