How to Build a Claude-Powered Document Q&A Tool

What a Claude document Q&A tool actually is

A document Q&A tool lets a user upload one or more files — PDFs, contracts, manuals, reports — and ask questions in plain English, getting answers grounded in that specific content instead of Claude's general training data. Think "what's the termination clause in this lease" or "summarize section 4 of this spec" with an answer that quotes the actual source text.

Building one comes down to four parts: getting clean text out of the document, finding the right portion of that text for a given question, sending that portion plus the question to Claude in a well-structured prompt, and returning the answer with enough context that users can verify it. None of this requires a custom model or fine-tuning — it's an engineering problem around retrieval and prompting, not a machine learning problem. The rest of this guide walks through each piece.

Architecture overview

A minimal pipeline looks like this:

1. Ingest: convert the uploaded file into plain text.
2. Chunk: split the text into manageable pieces (a few hundred to a couple thousand tokens each).
3. Retrieve: when a question comes in, pick the chunks most likely to contain the answer.
4. Generate: send the question and retrieved chunks to Claude with instructions to answer only from the provided text.
5. Return: stream the answer back and, ideally, show which chunk(s) it came from.

For short documents (a few pages) you can skip retrieval entirely and just send the whole document with the question — Claude's context window handles that comfortably. Retrieval only becomes necessary once documents get long enough that stuffing the whole thing into every request is wasteful or exceeds the context window.

Step 1: Extract text from the document

PDFs, Word docs, and scanned images all need different handling:

• PDF with real text: use a library like pdf-parse (Node) or pdfplumber (Python) to pull text directly.
• Scanned/image PDF: run OCR first (Tesseract or a cloud OCR service), then treat the output as plain text.
• Word/HTML: convert to plain text with mammoth or similar, stripping formatting.

Keep page numbers or section headers in the extracted text if you can — they're useful later for citations and for the model to reference specific locations.

Step 2: Chunk the text

Split extracted text into chunks that are small enough to retrieve selectively but large enough to preserve context — paragraph or section boundaries work better than fixed character counts because they avoid cutting sentences mid-thought. Store each chunk with metadata: source document, page/section, and chunk index. This step deserves its own careful implementation, especially for very long documents, but the short version is: chunk by structure, not by arbitrary length, and keep enough overlap between chunks that you don't lose context at the boundaries.

Step 3: Retrieve the relevant chunks

For a basic implementation, keyword or embedding-based search is enough:

// Simple embedding search (pseudocode)
const queryEmbedding = await embed(question);
const scored = chunks.map(chunk => ({
  ...chunk,
  score: cosineSimilarity(queryEmbedding, chunk.embedding)
}));
const topChunks = scored.sort((a, b) => b.score - a.score).slice(0, 5);

You don't need a dedicated vector database to start — an in-memory array works fine for tools handling a handful of documents per user. Move to a proper vector store only once you're indexing thousands of documents concurrently.

Step 4: Build the prompt and call Claude

The prompt should make clear that Claude must answer only from the supplied text and say so explicitly when the answer isn't there — this is the single biggest lever for reducing hallucinated answers in a document Q&A tool.

You are answering questions about a document. Use only the excerpts below.
If the answer isn't in the excerpts, say "The document doesn't cover this."
Cite the section or page number for each claim you make.

Excerpts:
[1] (page 3) "Either party may terminate this agreement with 30 days written notice..."
[2] (page 7) "Termination for cause requires no notice period..."

Question: What's the notice period for termination?

You can send this through any Claude-compatible endpoint. If you're using SubToAPI to route requests through your own application API key instead of managing raw Claude credentials, the call looks like:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "max_tokens": 600,
    "messages": [
      {"role": "user", "content": "<prompt with excerpts and question>"}
    ]
  }'

The full request format is in the messages docs, and the quickstart walks through getting an API key set up in a few minutes.

Step 5: Add citations and guardrails

Two things separate a usable document Q&A tool from a toy demo:

• Citations. Ask Claude to reference the chunk number or page in its answer, then map that back to a UI element — highlight the source paragraph, link to the page, or show a tooltip with the original text. This is what makes users trust the tool enough to act on its answers.
• Refusal handling. If the retrieved chunks don't contain the answer, Claude should say so rather than guessing. Test this explicitly with questions you know aren't answerable from the document and tune the prompt until it reliably declines.

Step 6: Stream the response

For longer answers, especially summaries or multi-part questions, streaming the response token by token makes the tool feel responsive instead of making users wait several seconds for a full answer to appear. SubToAPI supports streaming responses the same way — see the streaming docs for the event format and a working example.

Cost and scaling notes

Document Q&A tools can get expensive fast if every question re-sends the full document. Keep requests lean by only sending the chunks retrieval actually selected, cap max_tokens on the response to something reasonable for the expected answer length, and cache embeddings per document so you're not re-computing them on every question. If you're building this for multiple customers or teams, SubToAPI's plans give you per-key usage visibility so you can see which documents or users are driving cost, which matters once you move past a prototype into something people pay for.

Questions

Do I need a vector database to build this? Not for a first version. In-memory embedding search works fine for small document sets. Add a vector database once you're indexing thousands of documents or need persistence across restarts.

How do I stop Claude from making up answers not in the document? Be explicit in the prompt that it should only use the provided excerpts and should say when the answer isn't present, then test with unanswerable questions until the behavior is reliable.

Can this handle scanned PDFs, not just text PDFs? Yes, but you need an OCR step first to convert the scanned images into text before chunking and retrieval — Claude works with the extracted text, not the image itself.