How to Build a Knowledge Base Chatbot With Claude

A knowledge base chatbot built on Claude answers questions using your own documents — docs, FAQs, support tickets, internal wikis — instead of relying only on what the model learned during training. The core technique is retrieval-augmented generation (RAG): you search your content for the most relevant chunks, inject them into Claude's context window, and let the model generate an answer grounded in that text.

This guide walks through the full pipeline: preparing your content, retrieving relevant passages, constructing the prompt, calling the Claude API, and handling production concerns like streaming and citations. The architecture works whether you're using Claude directly or through a proxy like SubToAPI that gives you an application-level API key instead of a shared account.

The Basic Architecture

A knowledge base chatbot has four moving parts:

1. Ingestion — split your documents into chunks and store them with embeddings
2. Retrieval — on each user query, find the chunks most relevant to the question
3. Generation — send the question plus retrieved chunks to Claude with a system prompt that constrains it to the provided context
4. Delivery — stream the answer back to the user, optionally with citations

None of this requires fine-tuning. Claude's large context window means you can often pass several retrieved passages per query and get accurate, grounded answers.

Step 1: Chunk and Index Your Content

Split documents into chunks of roughly 300–800 tokens, keeping paragraphs or sections intact rather than cutting mid-sentence. Overlapping chunks by 10–15% helps preserve context across boundaries.

Store each chunk with:

• An embedding vector (from any embedding model)
• The source document ID and title
• A stable chunk ID for citation purposes

Any vector database works here — Postgres with pgvector, Pinecone, Weaviate, or even an in-memory index if your knowledge base is small. The choice of vector store is independent of which LLM you use for generation.

Step 2: Retrieve Relevant Chunks

On each incoming question, embed the query and run a similarity search against your index:

async function retrieve(query, topK = 5) {
  const queryEmbedding = await embed(query);
  const results = await vectorStore.search(queryEmbedding, topK);
  return results.map(r => ({
    text: r.chunkText,
    source: r.documentTitle,
    id: r.chunkId
  }));
}

Five chunks is a reasonable starting point. Fewer risks missing the right passage; more dilutes the context and increases token cost. If your knowledge base covers distinct domains (billing, technical docs, HR policy), consider filtering by category before the similarity search to improve precision.

Step 3: Build the Prompt

The system prompt is where you set the ground rules: answer only from the provided context, say "I don't know" when the context doesn't cover the question, and cite sources.

const systemPrompt = `You are a support assistant. Answer only using the
context below. If the answer isn't in the context, say you don't know
and suggest contacting support. Cite the source title in brackets after
any claim, like [Billing FAQ].

Context:
${chunks.map(c => `[${c.source}]\n${c.text}`).join('\n\n')}`;

Keep the context block tightly scoped to retrieved chunks rather than dumping your entire knowledge base — this keeps latency and cost predictable and reduces the chance of the model drifting off-topic.

Step 4: Call Claude

Once you have the system prompt and the user's question, send a standard messages request. If you're using SubToAPI, this looks like any other HTTPS call with your application key:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 1024,
    "system": "'"$SYSTEM_PROMPT"'",
    "messages": [
      {"role": "user", "content": "How do I reset my API key?"}
    ]
  }'

See /docs/quickstart for setup and /docs/messages for the full request and response schema. The benefit of routing through SubToAPI instead of raw provider credentials is that each app or environment gets its own sub_live_... key, so you can track usage per chatbot deployment without sharing a single account across your team.

Step 5: Stream the Response

Chatbots feel unresponsive if the user waits for the full answer before seeing anything. Stream tokens as they're generated:

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet-4-5",
    max_tokens: 1024,
    system: systemPrompt,
    messages: [{ role: "user", content: userQuestion }],
    stream: true
  })
});

for await (const chunk of response.body) {
  // forward each event to your frontend via SSE or WebSocket
  process.stdout.write(chunk);
}

Details on event formats are in /docs/streaming. Streaming also lets you show a "searching the knowledge base..." state while retrieval runs, then switch to token-by-token output once generation starts.

Step 6: Add Tool Use for Dynamic Lookups

Pure RAG works well for static content, but some questions need live data — account status, order history, current pricing tiers. For these, give Claude a tool instead of (or alongside) retrieved text:

{
  "name": "lookup_order_status",
  "description": "Get the current status of a customer order by ID",
  "input_schema": {
    "type": "object",
    "properties": { "order_id": { "type": "string" } },
    "required": ["order_id"]
  }
}

Claude decides when to call the tool versus answering from context. This hybrid pattern — static knowledge via retrieval, dynamic facts via tools — covers most real-world support and internal-assistant use cases. Full tool-calling examples are in /docs/tools.

Production Considerations

• Citations: return the source chunk IDs alongside the answer so your frontend can link back to the original document
• Fallback behavior: if retrieval returns no relevant chunks above a similarity threshold, skip the Claude call entirely and show a "no results" message — this saves cost and avoids hallucinated answers
• Usage visibility: if multiple team members or services call your chatbot's backend, separate API keys per environment make it easier to see which one is driving token usage — SubToAPI's dashboard shows this per key, which is useful once you have staging, production, and internal testing all hitting the same knowledge base
• Cost control: cap max_tokens, trim chunk count for simple questions, and consider a cheaper/faster model for first-pass retrieval relevance checks before the main generation call

Pricing for running this at scale, including per-seat team plans, is on /pricing, and you can start testing the flow with a free trial at /signup.

questions

Do I need a vector database to build a knowledge base chatbot with Claude? For anything beyond a handful of documents, yes. A vector store makes semantic search fast and scalable; for very small, static knowledge bases you could skip embeddings and pass the full text directly in the prompt.

How much of my knowledge base should I put in the context window? Only the retrieved chunks relevant to the current question — typically 3 to 8 chunks. Passing your entire knowledge base every call wastes tokens and makes it harder for Claude to focus on the right information.

Can the chatbot cite its sources? Yes. Instruct Claude in the system prompt to reference source titles or IDs for each claim, and store chunk metadata (document title, URL, section) at ingestion time so you can turn those citations into clickable links in your UI.