How to Build a Knowledge Base Chatbot (Step by Step)

A knowledge base chatbot answers user questions using your own documents — help center articles, internal wikis, product docs, PDFs — instead of relying purely on a model's general training. Building one comes down to three things: getting your content into a searchable format, retrieving the right chunks for each question, and sending those chunks plus the question to an LLM with instructions to answer only from what's provided.

This guide walks through the full pipeline: chunking your content, embedding and retrieval, prompt construction, and wiring it up to a model API with streaming so answers feel responsive. The examples use Claude via SubToAPI, but the architecture applies to any LLM backend.

The architecture in one picture

A knowledge base chatbot is a pipeline, not a single API call:

1. Ingest — pull in your docs (markdown, PDFs, HTML, Notion exports).
2. Chunk — split documents into passages small enough to retrieve precisely but large enough to carry context (usually 300–800 tokens).
3. Embed — convert each chunk into a vector using an embedding model.
4. Store — put vectors in a vector database (pgvector, Pinecone, Weaviate, or even an in-memory array for small KBs).
5. Retrieve — on each question, embed the query and fetch the top-k most similar chunks.
6. Generate — send the question + retrieved chunks to an LLM with a system prompt that constrains it to the provided context.

This is the standard RAG (retrieval-augmented generation) pattern. Skipping retrieval and just dumping your whole knowledge base into the prompt works for small KBs but breaks down past a few thousand tokens — you'll hit context limits, pay more per request, and get worse answers because the model has to sift through irrelevant content.

Step 1: Chunk your documents

Split long documents by headings or paragraphs rather than fixed character counts. Heading-aware chunking keeps related information together, which matters more for answer quality than hitting an exact token count.

function chunkByHeadings(markdown, maxTokens = 500) {
  const sections = markdown.split(/\n(?=#{1,3}\s)/);
  const chunks = [];
  for (const section of sections) {
    if (section.length / 4 <= maxTokens) {
      chunks.push(section);
    } else {
      // split oversized sections into paragraphs
      chunks.push(...section.split("\n\n"));
    }
  }
  return chunks.filter(c => c.trim().length > 20);
}

Store each chunk with metadata: source document, URL, and section title. You'll need this later to cite sources in the chatbot's answers.

Step 2: Embed and store

Generate embeddings for every chunk and store them alongside the raw text and metadata. For most teams, a managed vector database (pgvector on Postgres, or a hosted service) is simpler than running your own index. At query time, embed the user's question with the same embedding model and run a similarity search to get the top 3–6 chunks.

async function retrieveContext(question, topK = 5) {
  const queryVector = await embed(question);
  const results = await vectorDb.query({
    vector: queryVector,
    topK,
    includeMetadata: true,
  });
  return results.map(r => ({
    text: r.metadata.text,
    source: r.metadata.source,
  }));
}

Step 3: Build the prompt

The system prompt is where most knowledge base chatbots succeed or fail. You want the model to answer from the retrieved context, admit when it doesn't know, and avoid hallucinating facts that aren't in your docs.

const systemPrompt = `You are a support assistant. Answer the user's question
using ONLY the context below. If the answer isn't in the context, say
"I don't have that information" instead of guessing. Cite the source
title when relevant.

Context:
${retrievedChunks.map(c => `[${c.source}]\n${c.text}`).join("\n\n")}`;

Being explicit about the fallback behavior ("I don't have that information") is the single highest-leverage instruction in this prompt. Without it, models will confidently fill gaps with plausible-sounding but incorrect answers.

Step 4: Call the model with streaming

Once you have the system prompt and the user's question, send both to the model. Streaming the response matters for UX — users should see the answer appear token by token rather than waiting for the full response.

Here's a working example using SubToAPI, which exposes Claude through a standard HTTPS API with an sub_live_... key, so you don't need to manage separate LLM provider credentials for a side project or internal tool:

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    system: systemPrompt,
    max_tokens: 500,
    stream: true,
    messages: [{ role: "user", content: userQuestion }],
  }),
});

For the full streaming implementation, including how to parse server-sent events on the client, see the streaming docs. If you're starting fresh, the quickstart covers authentication and your first request end to end, and the messages endpoint reference documents all the request parameters.

Step 5: Add guardrails and citations

Two small additions dramatically improve trust in the chatbot:

• Citations: return the source metadata alongside the answer so users can click through to the original doc.
• Confidence fallback: if retrieval returns low-similarity chunks (below a threshold), skip generation entirely and show a "no good match found" message instead of letting the model guess.

if (topScore < 0.75) {
  return { answer: "I couldn't find a confident match for that question.", sources: [] };
}

Keeping it maintainable

As your knowledge base grows, re-index on a schedule (nightly or on doc publish) rather than per-request. Track which questions return no good matches — that list is your best source of content gaps to fill. And watch token usage: longer context windows mean higher cost per query, so tune topK and chunk size rather than defaulting to "retrieve everything."

If you're running this in production with multiple team members iterating on prompts, a dashboard with usage metadata per API key helps catch runaway costs early — something worth checking before you scale past a prototype. See pricing for plan details if you're evaluating options.

Questions

Do I need a vector database to build a knowledge base chatbot? Not for small knowledge bases (under a few hundred chunks) — you can compute cosine similarity in memory. Past that, a vector database like pgvector or Pinecone keeps retrieval fast and scalable.

How do I stop the chatbot from making up answers? Constrain the system prompt to only use the retrieved context, explicitly instruct a fallback response for missing information, and add a similarity-score threshold that skips generation when no good match is found.

What's the difference between a knowledge base chatbot and fine-tuning? Retrieval-augmented generation (RAG) keeps your data separate from the model and lets you update content instantly by re-indexing. Fine-tuning bakes knowledge into the model weights, which is slower to update and usually unnecessary for most support or docs use cases.