How to Build a Claude-Powered Knowledge Base Chatbot

A knowledge base chatbot answers questions using your own documentation, wikis, or support articles instead of relying on Claude's general training data. To build one, you need three pieces working together: a retrieval system that finds relevant content from your knowledge base, a prompt structure that feeds that content to Claude as context, and an API layer that handles the actual model calls reliably.

This is different from a generic chatbot because the goal isn't creativity — it's accuracy grounded in a specific document set. Below is a complete, practical architecture you can implement in an afternoon, along with the code to wire it up.

The architecture: retrieval-augmented generation (RAG)

The standard pattern for a knowledge base chatbot is RAG — retrieval-augmented generation. It works like this:

1. Ingest your documents and split them into chunks (paragraphs or sections, not whole files).
2. Embed each chunk into a vector using an embedding model, and store those vectors in a vector database.
3. Retrieve the most relevant chunks for a user's question by comparing the question's embedding against your stored vectors.
4. Generate an answer by sending the retrieved chunks plus the user's question to Claude as context.

Claude never "knows" your knowledge base ahead of time — it reads the relevant pieces fresh on every request. This keeps answers current and lets you update the knowledge base without retraining anything.

Step 1: Chunk and store your knowledge base

Break documents into chunks of roughly 300–800 tokens, ideally split at natural boundaries like headings or paragraphs rather than fixed character counts. Each chunk should carry metadata — source URL, title, last-updated date — so you can cite it later.

Store the chunks in a vector database (Pinecone, Weaviate, pgvector, or even a simple in-memory array for small datasets). You don't need Claude for this step — any embedding model works.

Step 2: Retrieve relevant chunks at query time

When a user asks a question, embed the question and run a similarity search against your stored chunks. Pull the top 3–6 matches. Don't over-retrieve — dumping 20 chunks into the prompt wastes tokens and often confuses the model with irrelevant content.

async function retrieveContext(question, vectorStore) {
  const queryEmbedding = await embed(question);
  const matches = await vectorStore.query(queryEmbedding, { topK: 5 });
  return matches.map(m => m.metadata.text).join("\n\n---\n\n");
}

Step 3: Build the grounded prompt

The prompt is where most knowledge base chatbots succeed or fail. Be explicit about two things: use only the provided context, and say when the answer isn't there.

const systemPrompt = `You are a knowledge base assistant. Answer the user's
question using ONLY the context below. If the context does not contain
the answer, say "I don't have that information in the knowledge base"
instead of guessing. Cite the source title when you use a fact.

Context:
${context}`;

This instruction matters more than model choice. Without it, Claude will happily fill gaps with plausible-sounding but unverified information.

Step 4: Call Claude to generate the answer

Once you have the context and the question, send a standard chat completion request. If you're using SubToAPI as your API layer on top of your existing Claude access, the call looks like this:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "max_tokens": 600,
    "system": "'"$systemPrompt"'",
    "messages": [
      {"role": "user", "content": "How do I reset a user'\''s password?"}
    ]
  }'

SubToAPI issues application-scoped sub_live_... keys, so your knowledge base service can call the API directly without routing requests through a personal Claude login. That's a meaningful difference once you move from a prototype to something running in production with real traffic. See the quickstart and the messages docs for the full request format.

Step 5: Stream the response for a better UX

Knowledge base answers can run long, especially with citations. Streaming the response token-by-token makes the chatbot feel responsive instead of making users stare at a blank screen for several seconds. The streaming docs cover the event format if you want to pipe responses directly into a chat UI with a typing effect.

Step 6: Add tool use for live lookups (optional)

If part of your knowledge base changes frequently — ticket status, pricing, inventory — static retrieval isn't enough. You can combine RAG with tool use so Claude calls a function to fetch live data when the static context doesn't cover it. A common pattern: try RAG first, and only invoke a tool call if the retrieved chunks don't contain an answer with sufficient confidence.

Keeping answers accurate

A few practices make a real difference in output quality:

• Chunk size matters. Too small and you lose context; too large and you waste tokens on irrelevant text.
• Always instruct the model to say "I don't know." This single line prevents most hallucinated answers.
• Re-rank before you generate. A cheap re-ranking step after vector search often improves relevance more than switching embedding models.
• Log every query and the chunks retrieved. When an answer is wrong, you need to know whether retrieval or generation failed.
• Refresh your vector store on a schedule. Stale embeddings from outdated docs are a common source of wrong answers.

Running it in production

Once the prototype works, the bottleneck is usually operational: rate limits, cost tracking per team, and giving non-engineers visibility into usage without sharing raw credentials. If you're already building on Claude, routing your knowledge base chatbot through SubToAPI gives you a dashboard for usage metadata, per-key limits, and team seats, rather than building that tooling yourself. Plans start at €9/month for solo projects and scale to team and multi-seat setups — see pricing or start with a free trial at signup.

Questions

Do I need a vector database to build this, or can I skip retrieval entirely? For small knowledge bases (under ~50 pages) you can sometimes paste the whole document set directly into the system prompt. Beyond that, retrieval becomes necessary both for accuracy and to stay within context limits.

How do I stop the chatbot from hallucinating when the knowledge base doesn't have an answer? Explicitly instruct the model in the system prompt to say it doesn't know rather than guess, and test this with questions you know aren't covered by your documents.

Can I add citations so users can verify answers? Yes — store a title and URL with each chunk's metadata, include that metadata in the prompt, and instruct Claude to reference the source title when it uses a fact from the context.