How to Build a Claude-Powered Knowledge Base Search

Building a Claude-powered knowledge base search means combining a retrieval system (usually vector search over your documents) with Claude's language understanding to answer questions in natural language, grounded in your own content. Instead of users scrolling through wikis, PDFs, or support docs, they ask a question and get a direct answer with sources — the classic retrieval-augmented generation (RAG) pattern, implemented with Claude as the reasoning layer.

This article walks through the full architecture: preparing your documents, indexing them for retrieval, querying Claude with the right context, and handling streaming and citations so the result feels like a real search experience rather than a chatbot bolted onto a help center.

The architecture in one picture

A Claude-powered knowledge base search system has four moving parts:

1. Ingestion — turn your docs, tickets, or wiki pages into clean text chunks.
2. Indexing — embed those chunks and store them in a vector database.
3. Retrieval — on each query, fetch the most relevant chunks.
4. Generation — send the query plus retrieved chunks to Claude and return a grounded answer.

Claude never "reads" your whole knowledge base at once. It only sees whatever you retrieve and inject into the prompt for that specific question. The quality of your search depends almost entirely on steps 1–3; Claude's job is just to synthesize a clear answer from good context.

Step 1: Prepare your content

Pull content from wherever it lives — Notion, Confluence, Markdown files, PDFs, a CMS — and normalize it into plain text with consistent metadata (title, URL, last updated date, section). Strip navigation chrome, footers, and boilerplate; they pollute embeddings and waste tokens.

Split documents into chunks of roughly 300–800 tokens, preserving semantic boundaries like headings and paragraphs rather than cutting mid-sentence. Store the source URL and section title alongside each chunk — you'll need it later for citations.

Step 2: Build the retrieval index

Generate embeddings for every chunk using any embedding model (OpenAI, Voyage, or a local model) and store them in a vector database — Pinecone, Weaviate, pgvector, or even a simple in-memory index for smaller knowledge bases. At query time:

const queryEmbedding = await embed(userQuestion);
const matches = await vectorDB.query({
  vector: queryEmbedding,
  topK: 5,
  includeMetadata: true,
});

Five to eight chunks is usually enough. Retrieving too many dilutes relevance and burns tokens; retrieving too few risks missing the answer. If your knowledge base is small (a few hundred pages), a hybrid approach — keyword search plus a lightweight re-ranker — often outperforms pure vector search for exact terms like product names or error codes.

Step 3: Query Claude with retrieved context

Once you have the top chunks, build a prompt that clearly separates instructions, retrieved context, and the user's question. Keep the system prompt strict about not inventing information that isn't in the context:

You are a knowledge base assistant. Answer only using the provided
context. If the answer isn't in the context, say you don't know and
suggest what the user could search for instead. Cite sources by
their [number] when you use them.

If you're already running your product on Claude, you can send this through SubToAPI's /v1/messages endpoint using your sub_live_ key instead of managing raw model credentials:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "max_tokens": 600,
    "system": "You are a knowledge base assistant. Answer only using the provided context. Cite sources by number.",
    "messages": [
      {
        "role": "user",
        "content": "Context:\n[1] Billing is monthly, charged on the 1st.\n[2] Refunds are processed within 5 business days.\n\nQuestion: When do refunds get processed?"
      }
    ]
  }'

SubToAPI exposes the same request/response shape documented at /docs/messages, so if you've used Claude directly before, the migration is just swapping the base URL and auth header — see /docs/quickstart for the full setup.

Step 4: Stream the answer

Search feels slow if users stare at a blank screen while Claude generates a few hundred tokens. Stream the response token by token and render it progressively, the same way modern search products do:

const res = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    max_tokens: 600,
    stream: true,
    messages: [{ role: "user", content: prompt }],
  }),
});

Full streaming setup, including handling server-sent events on the client, is covered in /docs/streaming.

Step 5: Handle citations and "I don't know"

Two things separate a trustworthy knowledge base search from a generic chatbot: citations and honest refusals.

• Citations: ask Claude to reference chunks by number in its answer, then map those numbers back to the source URLs you stored during indexing. Render them as clickable links under the answer.
• Refusals: if retrieval returns low-confidence matches (similarity scores below a threshold), skip the Claude call entirely and show a "no results found" state, or lower the bar for what counts as relevant. This avoids confident-sounding but wrong answers when the knowledge base simply doesn't cover the topic.

For more complex setups — where the assistant needs to call a function to fetch a live document, check permissions, or query a structured database instead of relying only on embeddings — tool use lets Claude decide when to call those functions mid-conversation. That's documented at /docs/tools.

Scaling and cost considerations

Knowledge base search tends to get noisy fast: every page load, every autocomplete keystroke, every "did you mean" retry can trigger a Claude call if you're not careful. A few practical guardrails:

• Cache answers for identical or near-identical queries for a short TTL.
• Only call Claude after the user submits a full query, not on every keystroke.
• Track token usage per query so you can spot runaway prompts (e.g., someone pasting a huge document into the search box).

If multiple team members or services need their own API access — one key for the internal search widget, one for a public-facing help center bot — SubToAPI lets you issue separate sub_live_ keys per use case and see usage broken down per key from the dashboard, which is useful once the project graduates from a prototype to something teams rely on daily. Plans start at €9/month on Solo, with team seat pricing at /pricing.

FAQ

Do I need a vector database to build this, or can I just send Claude my whole knowledge base? For anything beyond a handful of short documents, yes — stuffing everything into the context window is slow, expensive, and degrades answer quality. A vector index lets you retrieve only the relevant chunks per query.

How do I stop Claude from making up answers when the knowledge base doesn't cover a topic? Instruct it explicitly to say "I don't know" when the context is insufficient, and add a retrieval-confidence threshold that skips the Claude call entirely for low-relevance matches.

Can I use this pattern for internal docs and customer-facing help at the same time? Yes — index both sets of content separately (or tag chunks by audience) and route queries to the right index based on where the search is embedded, keeping the retrieval and prompting logic shared.