Claude API Document QA Pipeline Tutorial

A document QA pipeline lets users ask natural-language questions about PDFs, manuals, contracts, or internal wikis and get accurate answers grounded in the actual text — instead of a generic LLM response that might hallucinate. Building one with the Claude API involves three core stages: splitting documents into retrievable chunks, finding the chunks relevant to a question, and sending those chunks plus the question to Claude with a prompt that forces it to answer only from the provided context.

This tutorial walks through each stage with working code, then covers the parts people usually get wrong: chunk size, citation formatting, and how to stream the answer back to a frontend without blowing up your token budget. By the end you'll have a pipeline you can drop into a support tool, internal knowledge base, or customer-facing search feature.

Step 1: Chunk the documents

Claude handles long context well, but for QA pipelines you still want to split documents so you only send relevant material per query — it's cheaper and reduces noise. A simple, reliable approach is to chunk by paragraph or heading, with overlap so answers don't get cut mid-sentence.

function chunkText(text, maxChars = 1200, overlap = 150) {
  const chunks = [];
  let start = 0;
  while (start < text.length) {
    const end = Math.min(start + maxChars, text.length);
    chunks.push(text.slice(start, end));
    start += maxChars - overlap;
  }
  return chunks;
}

For most technical docs and contracts, 800–1500 character chunks with 100–200 character overlap works fine. Store each chunk with metadata: source filename, page number, and chunk index, so you can cite it later.

Step 2: Retrieve relevant chunks

You don't need a full vector database to get started. For small document sets (under a few hundred chunks), a lightweight embedding search is enough.

import { pipeline } from "@xenova/transformers";

const embedder = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2");

async function embed(text) {
  const output = await embedder(text, { pooling: "mean", normalize: true });
  return Array.from(output.data);
}

function cosineSim(a, b) {
  let dot = 0;
  for (let i = 0; i < a.length; i++) dot += a[i] * b[i];
  return dot;
}

function topChunks(queryEmbedding, chunkEmbeddings, chunks, k = 5) {
  return chunks
    .map((c, i) => ({ chunk: c, score: cosineSim(queryEmbedding, chunkEmbeddings[i]) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k)
    .map((r) => r.chunk);
}

At larger scale, swap this for a proper vector store (pgvector, Pinecone, Qdrant), but the retrieval logic stays the same: embed the question, compare against stored chunk embeddings, take the top-k.

Step 3: Build the prompt

This is the step that determines answer quality more than anything else. Give Claude the retrieved chunks as labeled context, instruct it to answer only from that context, and require it to cite sources.

function buildPrompt(question, chunks) {
  const context = chunks
    .map((c, i) => `[Source ${i + 1}: ${c.source}, p.${c.page}]\n${c.text}`)
    .join("\n\n---\n\n");

  return `You are answering questions using only the context below. If the answer isn't in the context, say you don't know — do not guess.

Context:
${context}

Question: ${question}

Answer the question and cite sources using [Source N] after each claim.`;
}

Step 4: Call the API

Once you have the prompt, send it as a user message. If you're using SubToAPI to expose your Claude access as a hosted HTTPS endpoint, the request looks like this:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-3-5-sonnet-20241022",
    "max_tokens": 800,
    "messages": [
      { "role": "user", "content": "<prompt from buildPrompt()>" }
    ]
  }'

The request/response shape follows the standard Messages API — full parameter reference is in /docs/messages, and /docs/quickstart walks through setting up your first key if you haven't already. The main advantage of routing through a service like this instead of raw provider credentials is centralized key management, per-app usage metadata, and team seats, which matters once more than one developer is touching the pipeline.

Step 5: Stream the answer

For anything user-facing, stream the response so the answer appears progressively rather than after a multi-second wait — especially important since document QA prompts are often large (full context chunks pushed in).

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-3-5-sonnet-20241022",
    max_tokens: 800,
    stream: true,
    messages: [{ role: "user", content: prompt }]
  })
});

for await (const chunk of response.body) {
  process.stdout.write(chunk.toString());
}

Streaming setup details, including SSE event types, are documented at /docs/streaming.

Handling edge cases

A few things that separate a working prototype from a production pipeline:

• No answer in context: explicitly instruct the model to say "I don't know" rather than guess — this is the single biggest hallucination fix.
• Conflicting sources: if two chunks contradict each other, ask Claude to flag the conflict instead of silently picking one.
• Large tables or code blocks: don't split mid-table; chunk at structural boundaries (headings, list items) rather than fixed character counts when the source format allows it.
• Multi-document questions: if a question spans multiple files, retrieve top-k per document rather than top-k globally, or one document will dominate the context.

Adding tool use for structured lookups

If your documents include structured data (pricing tables, API specs, config schemas), it's often more reliable to expose a lookup function as a tool rather than stuffing everything into the prompt. Claude can call the tool to fetch exact values instead of extracting them from free text. See /docs/tools for schema examples and how tool calls fit into the Messages flow.

Wrapping it into a service

Once the pipeline works end to end, wrap chunking, retrieval, and the Claude call behind a single API endpoint your frontend hits with just a question and a document ID. Keep the Claude API key server-side only. If you're running this across a team, /pricing has Solo, Team, and Scale plans with per-seat API keys and shared usage dashboards, which avoids everyone sharing one raw key in a shared .env file. You can test the full flow with a free trial at /signup before committing.

questions

Do I need a vector database to build a Claude document QA pipeline? No. For small document sets, in-memory embedding comparison (shown above) is enough. Move to pgvector, Pinecone, or Qdrant once you have thousands of chunks or need persistence across restarts.

How do I stop Claude from making up answers not in the document? Explicitly instruct it in the prompt to answer only from the provided context and say "I don't know" when the answer isn't present — this single instruction removes most hallucinations in practice.

What chunk size works best for QA accuracy? 800–1500 characters with 100–200 character overlap is a solid default for prose documents. Structured content (tables, code) should be chunked at natural boundaries instead of fixed lengths.