Build an Internal Knowledge Base Chatbot with Claude

If you want to build an internal knowledge base chatbot with Claude, the core problem isn't "how do I call an LLM" — it's how you get your company's documents, wikis, and tickets in front of the model at the right moment, and how you serve that reliably to a team without duct-taping API keys together.

This guide walks through the actual architecture: retrieval-augmented generation (RAG) over your internal docs, Claude as the reasoning and answering layer, tool use for live lookups, and the operational pieces (auth, streaming, usage tracking, team access) that separate a weekend prototype from something colleagues actually rely on.

The architecture in one picture

An internal knowledge base chatbot has four moving parts:

1. Ingestion — pull docs from Confluence, Notion, Google Drive, Slack exports, or a database into a consistent text format.
2. Indexing — chunk the text and store it in a vector store (pgvector, Pinecone, Qdrant, or even a simple SQLite + embeddings table for small teams).
3. Retrieval — on each user question, embed the query and fetch the top matching chunks.
4. Generation — send the retrieved chunks plus the question to Claude, and stream back an answer with citations.

Claude sits only in step 4, but it's the step users actually interact with, so its output quality, latency, and reliability define the product experience.

Step 1: Chunk and index your documents

Keep chunks small enough to be precise (300–800 tokens) but large enough to carry context. Store the source document title, URL, and section heading alongside each chunk — you'll want to cite them in the final answer.

function chunkDocument(doc) {
  const chunks = [];
  const paragraphs = doc.text.split("\n\n");
  let buffer = "";

  for (const p of paragraphs) {
    if ((buffer + p).length > 700) {
      chunks.push({ text: buffer, source: doc.title, url: doc.url });
      buffer = "";
    }
    buffer += p + "\n\n";
  }
  if (buffer) chunks.push({ text: buffer, source: doc.title, url: doc.url });
  return chunks;
}

Embed each chunk once and store it. Re-index only when a source document changes, not on every query.

Step 2: Retrieve relevant context

At query time, embed the user's question and do a similarity search against your vector store. Pull the top 5–8 chunks — more than that just adds noise and burns tokens.

const matches = await vectorStore.query(queryEmbedding, { topK: 6 });
const context = matches
  .map((m, i) => `[${i + 1}] Source: ${m.source}\n${m.text}`)
  .join("\n\n");

Step 3: Ask Claude to answer using only that context

This is the part most teams get wrong: they let the model answer freely instead of constraining it to the retrieved material. A good system prompt for an internal knowledge base bot explicitly forbids guessing.

You are an internal knowledge assistant. Answer only using the provided
context. If the answer isn't in the context, say you don't know and
suggest who to ask internally. Always cite sources using [1], [2], etc.

If you're routing requests through SubToAPI, the call to Claude looks like a normal messages request — you get an sub_live_... key from your dashboard and hit the same endpoint your app already uses for other Claude features:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 800,
    "system": "You are an internal knowledge assistant. Answer only using the provided context and cite sources.",
    "messages": [
      { "role": "user", "content": "Context:\n'"$CONTEXT"'\n\nQuestion: How do we handle refund disputes over $500?" }
    ]
  }'

See /docs/messages for the full request/response shape.

Step 4: Stream the answer for a real chat feel

Internal tools live or die on responsiveness. Streaming tokens as they're generated makes a multi-second retrieval-plus-generation flow feel instant instead of frozen.

const res = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-sonnet-4-5",
    max_tokens: 800,
    stream: true,
    messages: [{ role: "user", content: prompt }],
  }),
});

const reader = res.body.getReader();
// forward chunks to your frontend as they arrive

Streaming setup is covered in /docs/streaming — it works over standard SSE, so any frontend chat UI can consume it without extra libraries.

Step 5: Add tool use for live data

Static document retrieval covers wikis and PDFs, but internal chatbots often need to answer questions that require a live lookup — ticket status, deployment state, or an employee's team. Tool use lets Claude call a function you define instead of guessing.

{
  "tools": [
    {
      "name": "lookup_ticket",
      "description": "Fetch current status of an internal support ticket by ID",
      "input_schema": {
        "type": "object",
        "properties": { "ticket_id": { "type": "string" } },
        "required": ["ticket_id"]
      }
    }
  ]
}

When Claude decides it needs live data, it returns a tool call instead of text; your backend executes it and sends the result back in the next message. This combination — RAG for static knowledge, tool use for live systems — covers most internal chatbot requirements without needing a fine-tuned model. Full syntax is in /docs/tools.

Operational concerns people forget

• Access control: not every document should be searchable by every employee. Filter retrieval results by the requesting user's permissions before they ever reach the prompt.
• Answer auditing: log the retrieved chunks alongside the final answer so you can debug bad responses later.
• Team keys, not shared secrets: if multiple engineers or services call the API, give each one its own key rather than sharing one across the org. SubToAPI's Team and Scale plans include per-seat API keys and usage metadata specifically for this, so you can see which integration or teammate is generating traffic — see /pricing.
• Rate and cost visibility: internal bots tend to get heavy, bursty usage once people realize they work. Keep an eye on token usage per team, not just total spend.

Getting started quickly

You don't need to build the Claude-calling layer from scratch. Sign up at /signup, generate a sub_live_... key, and follow /docs/quickstart to get a working request-response loop in a few minutes — then plug in your retrieval pipeline around it.

questions

Do I need a vector database to build this, or can I skip it for a small team? For under a few hundred documents, a simple in-memory or SQLite-based embedding store is enough. Move to a dedicated vector database (pgvector, Qdrant) once you're past a few thousand chunks or need sub-second retrieval at scale.

Should the chatbot fine-tune on our internal documents instead of using RAG? No — RAG is cheaper, easier to update (just re-index changed docs), and lets you cite sources. Fine-tuning bakes knowledge into weights, making it hard to update and impossible to audit which document an answer came from.

How do I stop the bot from making up answers when the knowledge base doesn't have the information? Constrain the system prompt to only use provided context and explicitly instruct it to say "I don't know" when the retrieved chunks don't cover the question. Testing with deliberately unanswerable questions during development catches most hallucination issues early.