How to Build a Claude-Powered Internal Knowledge Base

2026-11-07 · 5 min read · SubToAPI Team

How to Build a Claude-Powered Internal Knowledge Base

Building a Claude-powered internal knowledge base means combining three things: a store of your company's documents, a retrieval step that finds the relevant chunks for a given question, and Claude generating an answer grounded in that retrieved content. This pattern is usually called retrieval-augmented generation (RAG), and it's the fastest way to turn scattered wikis, PDFs, Slack exports, and support tickets into a tool employees can actually query in plain English.

The core mechanics are straightforward: you embed your documents into a vector store, search that store at query time, inject the top matches into Claude's context window alongside the user's question, and let the model synthesize an answer with citations. The harder parts are picking the right chunking strategy, keeping answers scoped to what's actually in your docs (not hallucinated), and exposing this safely to a team without everyone needing direct model access. Below is a practical build order you can follow.

Step 1: Decide what "knowledge base" actually means

Before writing code, scope the content. Internal knowledge bases typically pull from:

Confluence, Notion, or Google Docs exports
PDFs (contracts, runbooks, onboarding guides)
Slack or Teams message history
Support ticket resolutions
Codebase READMEs and API docs

Pick one or two sources first. A knowledge base that answers "how do I request VPN access" reliably beats one that half-answers everything.

Step 2: Chunk and embed your documents

Split documents into chunks of roughly 300–800 tokens with some overlap, then generate embeddings for each chunk using any embedding model (OpenAI's text-embedding-3-small, Cohere, or a local model). Store the vectors in a vector database — pgvector, Pinecone, Weaviate, or even SQLite with a vector extension for smaller teams.

import { chunkText } from "./chunker.js";
import { embed } from "./embeddings.js";

const chunks = chunkText(documentText, { maxTokens: 600, overlap: 80 });
for (const chunk of chunks) {
  const vector = await embed(chunk.text);
  await vectorStore.insert({
    text: chunk.text,
    vector,
    source: documentId,
  });
}

Keep metadata (source document, section, last-updated date) attached to each chunk — you'll need it for citations and for filtering out stale content later.

Step 3: Retrieve relevant chunks at query time

When a user asks a question, embed the query, run a similarity search, and pull back the top 5–10 chunks. Don't dump your entire knowledge base into the prompt — it's slower, more expensive, and often leads to Claude answering from the noisiest chunk instead of the most relevant one.

const queryVector = await embed(userQuestion);
const matches = await vectorStore.search(queryVector, { topK: 6 });
const context = matches.map(m => `Source: ${m.source}\n${m.text}`).join("\n\n");

Step 4: Build the prompt and call Claude

Construct a system prompt that instructs Claude to answer only from the provided context and to say when it doesn't know. This single instruction does more to prevent hallucinated answers than anything else in the pipeline.

const systemPrompt = `You are an internal knowledge assistant. Answer only using the
context below. If the answer isn't in the context, say you don't have that
information and suggest who to ask. Always cite the source document.`;

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    system: systemPrompt,
    max_tokens: 1024,
    messages: [
      { role: "user", content: `Context:\n${context}\n\nQuestion: ${userQuestion}` },
    ],
  }),
});

const data = await response.json();
console.log(data.content[0].text);

This is also where SubToAPI is useful if you're already a Claude subscriber: instead of signing up for a separate API billing account, you turn your existing access into an HTTPS endpoint with its own sub_live_... key, so your internal tool just calls a normal REST API. See the quickstart for the setup, and the messages docs for the full request format.

Step 5: Stream answers for a better UX

For a chat-style internal tool, streaming responses token-by-token makes the assistant feel responsive instead of making users stare at a spinner for several seconds. If you're building this on SubToAPI, enable stream: true and consume server-sent events — details are in the streaming guide.

Step 6: Add tool use for live lookups

Static documents go stale. For things like "who's on call this week" or "what's the status of ticket #4821," wire up a tool that queries your PagerDuty or Jira API directly instead of relying only on indexed documents. Claude can decide when to call the tool versus answer from retrieved context — see the tools documentation for the function-calling format.

{
  "tools": [
    {
      "name": "lookup_ticket",
      "description": "Fetch current status of a support ticket by ID",
      "input_schema": {
        "type": "object",
        "properties": { "ticket_id": { "type": "string" } },
        "required": ["ticket_id"]
      }
    }
  ]
}

Step 7: Control who can access it

An internal knowledge base usually needs team-level access control — engineering shouldn't necessarily see HR's docs, and you want usage visibility per department. Rather than sharing one raw API key across the company, issue separate application keys per team or project and track usage per key from a dashboard. This is one of the reasons teams use SubToAPI instead of a shared key: seats and billing scale with the team (pricing), and every key's usage is visible without building your own logging layer.

Step 8: Keep the index fresh

Set up a nightly or webhook-triggered re-index job for any document that changes. A knowledge base answering from six-month-old onboarding docs is worse than no knowledge base at all, because it erodes trust fast.

Putting it together

The full pipeline is: ingest → chunk → embed → retrieve → prompt → generate → (optionally) tool-call → stream to the user. None of these steps individually is hard; the value comes from doing all of them consistently and keeping the index current. Start with a narrow source (one wiki space, one doc folder), get the retrieval quality right, then expand scope once people trust the answers.

If you want to skip separate API billing setup and reuse your existing Claude subscription for this, sign up and get the endpoint running in a few minutes with a free trial.

questions

Do I need a vector database to build this? Yes, for anything beyond a handful of documents. A vector store is what makes retrieval fast and relevant; without it you'd have to send your entire document set to Claude on every query, which is slow and expensive.

Can Claude read PDFs and Word documents directly? Claude can process document content, but for a knowledge base you still want to extract and chunk the text yourself first — this gives you control over citations, metadata, and which parts of a document get retrieved.

How do I stop the assistant from making up answers? Instruct it explicitly to answer only from the provided context and to say when information isn't present, and keep retrieval precise (top 5–10 relevant chunks) rather than dumping the whole knowledge base into every prompt.

How to Build a Claude-Powered Internal Knowledge Base