How to Build a Claude-Powered Document QA Tool

Building a Claude-powered document QA tool means combining a retrieval layer — something that finds the relevant chunks of text inside your documents — with Claude's reasoning to answer questions grounded in that content. The core pattern is always the same: split documents into chunks, embed and index them, retrieve the top matches for a user's question, then pass those chunks to Claude as context so it answers only from what's actually in the document instead of guessing.

This article walks through the full pipeline: chunking strategy, retrieval, prompt design for grounded answers, citation handling, and how to expose the whole thing as a production API endpoint your frontend or internal tools can call.

The Architecture at a Glance

A document QA tool has four moving parts:

1. Ingestion — parse PDFs, docs, or text into clean plain text.
2. Chunking + indexing — split text into overlapping chunks and embed them in a vector store (or use a simpler keyword search if your corpus is small).
3. Retrieval — on each question, fetch the top N relevant chunks.
4. Generation — send the question + retrieved chunks to Claude and return a grounded answer, ideally with citations back to the source.

None of this requires a heavyweight framework. You can build a working prototype with a chunking function, any vector database (Pinecone, Qdrant, pgvector, or even a local FAISS index), and direct API calls to Claude.

Step 1: Chunk Your Documents

Chunk size matters more than people expect. Too small and Claude loses context; too large and retrieval gets noisy and you waste tokens. A good default is 500–800 tokens per chunk with roughly 15% overlap so sentences aren't cut mid-thought.

function chunkText(text, chunkSize = 600, overlap = 100) {
  const words = text.split(/\s+/);
  const chunks = [];
  let i = 0;
  while (i < words.length) {
    const chunk = words.slice(i, i + chunkSize).join(" ");
    chunks.push(chunk);
    i += chunkSize - overlap;
  }
  return chunks;
}

Store each chunk with metadata: document name, page number if available, and a chunk ID. You'll need these later to build citations.

Step 2: Embed and Retrieve

Embed each chunk with your embedding model of choice and store the vectors alongside the metadata. At query time, embed the user's question and run a similarity search to pull back the top 4–8 chunks. For smaller document sets (under a few hundred pages), a simple BM25 or keyword search can outperform embeddings for exact-term questions like "what's the termination clause," so it's worth testing both or combining them (hybrid search).

Don't over-engineer retrieval before you've tested with real questions. Start with top-k similarity search, measure answer quality, then tune.

Step 3: Prompt Claude to Answer Only From Context

The prompt is where most document QA tools fail. If you just paste chunks and ask a question, Claude will sometimes blend in outside knowledge. Be explicit that it should answer strictly from the provided context and say when it doesn't know.

You are a document assistant. Answer the user's question using ONLY the
context below. If the answer is not contained in the context, say
"I couldn't find that in the provided documents."

Context:
[CHUNK_1]
[CHUNK_2]
[CHUNK_3]

Question: {user_question}

Answer, then list which chunk numbers you used as sources.

Asking Claude to cite which chunks it used is cheap and effective — it gives you a built-in citation mechanism without extra tooling, and it makes hallucinations easier to catch because you can check the cited chunk against the answer.

If you want structured citations instead of free text, Claude's tool use feature lets you define a function schema that forces the model to return {answer, sources: [chunk_id, ...]} as structured JSON instead of prose, which is easier to render in a UI. See /docs/tools for how tool calling works.

Step 4: Call the Model and Stream the Response

For a chat-style document QA UI, streaming matters — users expect the answer to appear token by token rather than waiting several seconds for a full response. If you're routing requests through SubToAPI instead of managing raw provider credentials, the call looks like this:

const res = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    stream: true,
    max_tokens: 1024,
    messages: [
      { role: "user", content: promptWithContext }
    ]
  })
});

Streaming is handled the same way as any Claude Messages call — see /docs/streaming for the event format if you're parsing server-sent events on the client.

Step 5: Ship It Behind a Real API

A document QA tool usually starts as an internal script but ends up needing to be a service other apps or team members call — a support dashboard, a Slack bot, a customer-facing help widget. At that point you need API keys, usage tracking per team or per app, and rate limiting, none of which is worth building yourself for a side project or an MVP.

This is exactly the gap SubToAPI fills: it turns your Claude access into a standard HTTPS API with application-scoped keys (sub_live_...), streaming, tool use, and usage metadata per key, so you can give your document QA tool its own key, your internal admin tool another, and see usage broken down by application in one dashboard. Setup takes a few minutes — see /docs/quickstart — and the Messages endpoint maps directly to the pattern described above, so moving from a local prototype to a deployed API doesn't require rewriting your prompt logic. Check /pricing if you're deciding between Solo, Team, and Scale tiers, or just start a free trial at /signup.

Common Pitfalls

• Chunking without overlap — answers that span two chunks get silently truncated.
• Too many chunks in context — dumping 20 chunks into the prompt dilutes relevance and increases cost. Stick to the top 4–8 unless you have evidence more helps.
• No "I don't know" instruction — without it, Claude will often fill gaps with plausible-sounding but unsupported text.
• Ignoring document updates — if source documents change, stale chunks in your index produce wrong answers. Re-index on document update, not on a fixed schedule.

questions

Do I need a vector database to build a document QA tool? Not necessarily. For small document sets, keyword search or even full-document context (if it fits Claude's context window) can work fine. A vector database becomes worth it once you have hundreds of documents or need fast semantic retrieval.

How do I stop Claude from hallucinating answers not in the document? Explicitly instruct the model to answer only from provided context and to say when it doesn't know, then ask it to cite the chunk IDs it used. This catches most hallucinations because you can verify the cited chunk actually supports the answer.

Can I add this to an existing app without managing API infrastructure myself? Yes — route your Messages calls through SubToAPI with a scoped API key per application, which gives you streaming, tool use, and usage metadata without building key management or rate limiting from scratch. See /docs/messages for the request format.