How to Build a Claude-Powered Knowledge Base Chatbot

Building a Claude-powered knowledge base chatbot means combining a retrieval layer over your documentation, wiki, or support articles with Claude's reasoning to answer questions conversationally, in context, and with citations. The core pattern is retrieval-augmented generation (RAG): you search your knowledge base for relevant chunks, inject them into Claude's context window, and let Claude answer using only that material.

This article walks through the actual architecture: how to chunk and index your content, how to retrieve the right pieces at query time, how to structure the system prompt so Claude stays grounded and cites sources, and how to wire it all into a chat interface with streaming and conversation memory. It assumes you already have a knowledge base (docs site, Notion export, PDFs, Confluence, support tickets) and want a chatbot layer on top.

Architecture Overview

A knowledge base chatbot has four moving parts:

1. Ingestion pipeline — pulls content from your source, cleans it, and splits it into chunks.
2. Vector store — stores embeddings of each chunk for semantic search.
3. Retrieval step — on each user query, fetches the most relevant chunks.
4. Claude call — receives the user's question, the retrieved chunks, and conversation history, and generates an answer.

Only the last step actually involves the Claude API. Everything else is plumbing you control. Get the plumbing right and the chatbot quality mostly takes care of itself.

Step 1: Chunk Your Content Properly

Chunking determines whether retrieval works at all. Too large and you waste context on irrelevant text; too small and you lose the surrounding meaning Claude needs to answer correctly.

Practical rules that hold up in production:

• Split by semantic boundaries (headings, paragraphs) rather than fixed character counts when possible.
• Target 300–800 tokens per chunk — enough for a complete thought, small enough to retrieve precisely.
• Keep metadata with each chunk: source title, URL, section heading, last-updated date. You'll need this for citations.
• Overlap chunks by 10–15% so answers that span a chunk boundary aren't lost.

function chunkDocument(text, maxTokens = 500, overlap = 50) {
  const paragraphs = text.split(/\n\n+/);
  const chunks = [];
  let current = [];
  let tokenCount = 0;

  for (const para of paragraphs) {
    const paraTokens = Math.ceil(para.length / 4);
    if (tokenCount + paraTokens > maxTokens && current.length) {
      chunks.push(current.join("\n\n"));
      current = current.slice(-1); // crude overlap
      tokenCount = 0;
    }
    current.push(para);
    tokenCount += paraTokens;
  }
  if (current.length) chunks.push(current.join("\n\n"));
  return chunks;
}

Step 2: Embed and Index

Any embedding provider plus a vector store (Postgres with pgvector, Pinecone, Weaviate, or even an in-memory array for small knowledge bases) works here. The key decision is re-indexing frequency — if your docs change daily, run ingestion on a schedule or trigger it on publish events rather than indexing once and forgetting it.

For knowledge bases under a few thousand chunks, don't overengineer this. A Postgres table with a vector column and a cosine similarity query is enough.

Step 3: Retrieve Relevant Chunks

At query time, embed the user's question and fetch the top 4–8 matching chunks. Two things improve retrieval quality beyond plain vector search:

• Hybrid search: combine vector similarity with keyword/BM25 search. Pure semantic search misses exact product names, error codes, or acronyms.
• Re-ranking: if your top-k results look noisy, pull 20 candidates and re-rank with a cheaper model or heuristic before sending the final 5–8 to Claude.

Step 4: Build the Claude Prompt

This is where answer quality is actually won or lost. The system prompt needs to do three jobs: define the chatbot's role, constrain it to the retrieved content, and instruct it to cite sources.

const systemPrompt = `You are a support assistant for Acme Docs.
Answer the user's question using ONLY the context below.
If the answer isn't in the context, say you don't know and suggest
contacting support — do not guess or use outside knowledge.
Cite sources using [1], [2] etc. matching the numbered context blocks.

Context:
[1] ${chunk1.title} — ${chunk1.text}
[2] ${chunk2.title} — ${chunk2.text}
[3] ${chunk3.title} — ${chunk3.text}`;

Send this as the system message, and pass the full conversation history as user/assistant turns so Claude can handle follow-up questions ("what about on mobile?") without re-explaining context.

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "system": "'"$SYSTEM_PROMPT"'",
    "messages": [
      {"role": "user", "content": "How do I reset my API key?"}
    ],
    "max_tokens": 600
  }'

If you're testing this quickly without managing your own Anthropic account and billing, SubToAPI exposes Claude through a standard HTTPS endpoint using your existing Claude access — useful if you want to prototype the chatbot backend without setting up separate API billing. See the Messages API docs for the full request shape.

Step 5: Stream Responses and Keep Conversation State

Users expect chat UIs to feel live. Stream tokens as they generate rather than waiting for the full response — see the streaming docs for the event format. On the state side, store conversation history per session (even just in memory for short sessions, or a database row keyed by session ID for persistence) and truncate older turns once you approach context limits, keeping the retrieved chunks fresh on every new question rather than reusing stale ones.

Step 6: Add Citations to the UI

Since each context block is numbered, parse [1], [2] references out of Claude's response text and render them as clickable links back to the source chunk's metadata (title + URL). This single feature does more for user trust than any prompt tweaking — people verify AI answers against sources far more than they argue with phrasing.

Common Failure Modes

• Hallucinated answers when retrieval finds nothing relevant. Fix: have your retrieval step return a confidence threshold; if the top match score is too low, skip the Claude call and tell the user directly that nothing matched.
• Stale answers after docs change. Fix: re-index on publish, not on a weekly cron.
• Context window bloat from long conversation history. Fix: summarize older turns into a single system note after 8–10 exchanges.
• Rate limits during traffic spikes. If you're running this behind a shared Claude account, check pricing for plans with higher throughput before you hit a wall in production.

Questions

Do I need a vector database to build this? Not strictly — for small knowledge bases (a few hundred pages), a Postgres table with pgvector or even an in-memory cosine similarity search is enough. Dedicated vector databases matter at scale or when you need hybrid search and filtering.

How many chunks should I send to Claude per query? Start with 4–8 chunks of 300–800 tokens each. Too few risks missing the answer; too many dilutes relevance and increases cost. Tune based on how often answers reference content outside the retrieved set.

How do I stop the chatbot from answering outside the knowledge base? Constrain the system prompt explicitly ("answer only using the context below") and set a retrieval confidence threshold below which you return a fallback message instead of calling Claude at all.