Building a Claude API Document Question Answering System

A document question answering system takes a file (or set of files) and lets users ask natural-language questions about its content, getting back accurate answers grounded in the source text. With the Claude API you can build this without a separate retrieval stack for small-to-medium document sets, because Claude's large context window lets you pass whole documents directly into the prompt instead of chunking and embedding everything first.

This article walks through the two main architectures — direct context stuffing and retrieval-augmented generation (RAG) — and shows working code for both, plus the practical details that make the difference between a demo and a production system: citation handling, token budgeting, and streaming responses back to the UI.

Two Architectures, One Decision Point

Direct context (long-context QA): You pass the full document (or documents) as part of the prompt, then ask your question. Claude reads everything and answers from what it sees. This works well when your documents fit comfortably inside the context window and you don't need to search across thousands of files.

RAG (retrieval-augmented generation): You split documents into chunks, embed them, store them in a vector database, retrieve the top-k relevant chunks for a given question, and only pass those chunks to Claude. This is the right approach once you have more content than fits in context, or when you need sub-second retrieval across a large corpus.

The decision point is simple: if your corpus per query is under roughly 100–150 pages, direct context is usually simpler, cheaper to build, and more accurate because nothing gets lost in the retrieval step. Beyond that, you need RAG.

Direct Context QA with Claude

The simplest version of this system is a single API call with the document embedded in the user message.

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 1024,
    "system": "Answer questions using only the provided document. If the answer is not in the document, say so explicitly. Quote the relevant sentence before answering.",
    "messages": [
      {
        "role": "user",
        "content": "DOCUMENT:\n<full text of the document here>\n\nQUESTION: What is the notice period for terminating this contract?"
      }
    ]
  }'

The system prompt does the heavy lifting here. Telling Claude to quote the source sentence before answering reduces hallucination significantly and gives users a way to verify the answer against the original text.

For multi-turn document QA — where users ask follow-up questions — keep the document in the system prompt (or the first user message) and append only new questions to the conversation, rather than re-sending the full document every turn. This keeps your token usage predictable across a session. See /docs/messages for how multi-turn conversations are structured.

Chunking and Retrieval for Larger Corpora

Once you're querying across many documents, a basic RAG pipeline looks like this:

async function answerQuestion(question, vectorStore) {
  const relevantChunks = await vectorStore.search(question, { topK: 5 });

  const context = relevantChunks
    .map((c, i) => `[${i + 1}] (source: ${c.filename})\n${c.text}`)
    .join("\n\n");

  const response = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "content-type": "application/json"
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-5",
      max_tokens: 1024,
      system: "Answer using only the numbered excerpts below. Cite the excerpt number for every claim.",
      messages: [
        { role: "user", content: `${context}\n\nQUESTION: ${question}` }
      ]
    })
  });

  return response.json();
}

The embedding and vector search step is independent of Claude — you can use any embedding model and vector store (pgvector, Pinecone, Qdrant, etc). Claude's job is strictly the final reasoning and answer generation step, which is why citation prompting matters: it forces the model to tie every statement back to a retrieved chunk instead of inventing one.

Streaming the Answer

For document QA inside a chat-style UI, stream the response so users see the answer forming instead of waiting for the full generation:

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "content-type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet-4-5",
    max_tokens: 1024,
    stream: true,
    messages: [{ role: "user", content: `${context}\n\nQUESTION: ${question}` }]
  })
});

Details on handling the event stream are covered in /docs/streaming — the short version is you read server-sent events and append text deltas to the UI as they arrive.

Handling Token Budgets

Document QA systems fail in production mostly because of token budget mismanagement, not model quality. A few rules that hold up:

• Reserve at least 20% of your context window for the system prompt, conversation history, and the model's answer — don't fill the window entirely with document text.
• Truncate or summarize documents that exceed the window rather than silently cutting them off mid-sentence.
• Track input and output token usage per request so you can catch runaway costs from oversized documents before they hit your bill.

If you're running this system for a team or across multiple apps, routing all of it through a single API layer makes usage tracking and key management far simpler than juggling individual Claude accounts. SubToAPI turns your existing Claude access into application API keys with per-key usage metadata, so you can see exactly which document QA endpoint or team member is driving token consumption. Check /pricing for plan details or /docs/quickstart to get your first key working in minutes.

Evaluating Answer Quality

Before shipping, test your system against a held-out set of question/answer pairs where you know the ground truth. Check specifically for:

• Faithfulness — does the answer only contain information present in the source document?
• Completeness — does it cover the full answer, or stop after the first matching sentence?
• Refusal correctness — does it say "not found in document" when the answer genuinely isn't there, instead of guessing?

A system that refuses to answer appropriately is far more trustworthy than one that always produces a confident-sounding response.

Questions

Does Claude need the document split into chunks to answer questions about it? No, not if the document fits in the context window. For documents under roughly 100-150 pages, pasting the full text directly into the prompt is simpler and more accurate than chunking.

How do I stop Claude from hallucinating answers not in the document? Instruct it explicitly to answer only from the provided text, require it to quote or cite the source passage, and tell it to say "not found" when the answer isn't present.

Can I build a multi-document QA system without a vector database? Yes, as long as the combined size of the documents you need per query stays within the context window. Once your corpus grows beyond that, you'll need chunking and retrieval.