Claude API Knowledge Base Question Answering Guide

Building question answering over a knowledge base with the Claude API means combining a retrieval step (finding the right documents) with a generation step (asking Claude to answer using only those documents). Claude doesn't automatically know your internal docs, support tickets, or product manuals — you have to feed the relevant text into the prompt at request time, then instruct the model to answer strictly from that context.

The short version: you chunk your documents, store them somewhere searchable (a vector database, full-text search, or even a simple keyword index for small datasets), retrieve the top matches for a given question, and pass those matches plus the question to Claude in a structured prompt. Claude then generates an answer grounded in the retrieved text, ideally with citations back to the source chunks. This pattern is usually called RAG (retrieval-augmented generation), and it's the standard approach for knowledge base Q&A whether you're building internal docs search, a support bot, or a product assistant.

Why you can't just paste the whole knowledge base in

Claude's context window is large, but "large" isn't the same as "unlimited and cheap." If your knowledge base is a handful of pages, you can paste everything into the system prompt and skip retrieval entirely — this works and it's simple. But once you're dealing with hundreds of documents, pasting everything on every request:

• Costs more tokens per call, every single time, even for unrelated questions
• Dilutes relevance — Claude has to find the needle in a much bigger haystack
• Makes it harder to cite exactly which document an answer came from

Retrieval solves this by narrowing the context down to the 3–10 chunks most likely to contain the answer, before Claude ever sees the question.

Chunking your knowledge base

Split documents into chunks of roughly 300–800 tokens, ideally along natural boundaries (headings, paragraphs, list items) rather than fixed character counts. Each chunk should carry metadata: source document, title, section, and a unique ID you can reference later for citations.

{
  "id": "docs-billing-042",
  "source": "billing-faq.md",
  "section": "Refund policy",
  "text": "Refunds are issued within 5 business days..."
}

For retrieval, generate embeddings for each chunk and store them in a vector database (Pinecone, pgvector, Weaviate, etc.), or use a hybrid approach combining keyword search with embeddings for better recall on exact terms like product names or error codes.

Structuring the prompt for grounded answers

Once you've retrieved the top chunks for a question, the prompt structure matters as much as the retrieval quality. A reliable pattern:

System: You are a support assistant. Answer only using the
provided context. If the answer isn't in the context, say
you don't know. Cite the source section for each claim.

User: Context:
[1] (Refund policy) Refunds are issued within 5 business days...
[2] (Billing cycles) Invoices are generated on the 1st of each month...

Question: How long does a refund take?

This structure does three things: it limits the model to the retrieved facts, it gives it labeled sources to cite, and it explicitly allows "I don't know" as an answer — which matters a lot for knowledge base Q&A, where a confident wrong answer is worse than no answer.

Here's a minimal example calling the Claude API via SubToAPI, which exposes the same message format behind your own API key:

const res = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    max_tokens: 500,
    system: "Answer only using the provided context. Cite sources by [id]. If not covered, say you don't know.",
    messages: [
      { role: "user", content: `Context:\n${retrievedChunks}\n\nQuestion: ${userQuestion}` }
    ]
  })
});

If you're new to the API shape, the quickstart and messages docs cover request/response formats in full.

Handling multi-document and follow-up questions

Real knowledge base Q&A rarely stops at one question. Users ask follow-ups that reference earlier answers ("what about for annual plans?"), which means you need conversation history, not just the latest question, feeding into retrieval. Two practical approaches:

1. Rewrite the follow-up into a standalone question using Claude itself before retrieval, so "what about annual plans" becomes "what is the refund policy for annual plans," then retrieve based on that.
2. Keep the last 2–3 turns in the prompt alongside fresh retrieved context, so Claude has both the conversation thread and the grounding documents.

For long-running sessions, streaming responses token-by-token keeps the UI responsive while Claude works through longer, multi-source answers — important when a question spans several retrieved chunks and the model needs to synthesize across them.

Adding tool use for live retrieval

Instead of doing retrieval entirely outside the API call, you can give Claude a search tool and let it decide when to query your knowledge base mid-conversation. This is useful when questions are ambiguous and the model benefits from issuing its own refined search query rather than relying on your initial retrieval pass.

{
  "name": "search_knowledge_base",
  "description": "Search the internal knowledge base for relevant document chunks",
  "input_schema": {
    "type": "object",
    "properties": { "query": { "type": "string" } },
    "required": ["query"]
  }
}

Claude calls the tool with its own query, you run the search, return results, and Claude continues reasoning with that data before producing a final answer. The tools documentation covers the full request/response loop for this pattern.

Running this in production

A few things that matter once knowledge base Q&A moves past a prototype:

• Track token usage per query so you know your retrieval chunk count isn't silently inflating costs — SubToAPI returns usage metadata on every response so you can monitor this per endpoint or per customer.
• Version your knowledge base — when docs change, re-embed and re-index, and tag answers with the index version for debugging stale answers.
• Rate limit and key scope per team — if multiple teams or customers query the same knowledge base through your app, issue separate application keys (sub_live_...) so you can see usage and set limits independently from the dashboard, without managing separate Claude accounts. See pricing for plan details, or start with a free trial at signup.

questions

Do I need a vector database for Claude API knowledge base Q&A? Not always. For small knowledge bases (under a few hundred chunks), keyword search or even including full documents in context can work fine. Vector search becomes worthwhile once relevance matters more and the dataset grows beyond what fits comfortably in a single prompt.

How do I stop Claude from making up answers not in my docs? Explicitly instruct it in the system prompt to answer only from the provided context and to say "I don't know" when the context doesn't cover the question. Grounding instructions combined with retrieved, relevant chunks significantly reduce fabricated answers.

Can I cite sources in the generated answer? Yes — give each retrieved chunk a labeled ID in the prompt and instruct Claude to reference those IDs when making claims. This lets your app map citations back to the original document for display to the user.