How to Build a Knowledge Base Search Tool

A knowledge base search tool lets users type a question in plain language and get a relevant answer pulled from your internal docs, help center, or wiki — instead of scrolling through a list of keyword-matched articles. The core technique is retrieval-augmented generation (RAG): you search your own content for relevant chunks, then hand those chunks to an LLM that writes a direct answer with citations.

This article walks through the full pipeline: ingesting documents, chunking them, generating embeddings, running semantic search, and using Claude to turn the retrieved text into a readable answer. By the end you'll have a working architecture you can adapt to any content source — Notion, Confluence, Markdown files, or a database of support tickets.

Why Keyword Search Isn't Enough

Traditional full-text search (like SQL LIKE or Elasticsearch's default BM25) matches literal words. If a user searches "how do I cancel my plan" but your doc says "terminating a subscription," keyword search misses it entirely. Semantic search solves this by comparing meaning, not just text, using vector embeddings.

A good knowledge base search tool combines two layers:

1. Retrieval — find the most relevant chunks of content using vector similarity.
2. Generation — feed those chunks to an LLM to produce a direct, synthesized answer instead of a list of links.

Step 1: Chunk Your Content

Before you can search anything, break your documents into chunks of 200–500 tokens. Chunking too large wastes context and dilutes relevance; too small loses surrounding context.

function chunkText(text, maxTokens = 400) {
  const sentences = text.split(/(?<=[.?!])\s+/);
  const chunks = [];
  let current = "";

  for (const sentence of sentences) {
    if ((current + sentence).length > maxTokens * 4) {
      chunks.push(current.trim());
      current = "";
    }
    current += sentence + " ";
  }
  if (current.trim()) chunks.push(current.trim());
  return chunks;
}

Store each chunk with metadata: source document ID, title, URL, and section heading. You'll need this later to show citations.

Step 2: Generate Embeddings

Each chunk needs a vector embedding so you can compare it mathematically to a user's query. Use any embedding API (OpenAI's text-embedding-3-small, Cohere, or a self-hosted model) and store the vectors in a vector database like Pinecone, Weaviate, or pgvector on Postgres.

const vector = await embeddingClient.embed(chunk.text);
await vectorDB.upsert({
  id: chunk.id,
  values: vector,
  metadata: { title: chunk.title, url: chunk.url }
});

Re-run this ingestion pipeline whenever source documents change, ideally on a scheduled job or a webhook from your CMS.

Step 3: Search and Retrieve

When a user submits a query, embed the query itself and run a similarity search against your vector store. Retrieve the top 5–8 chunks.

const queryVector = await embeddingClient.embed(userQuery);
const results = await vectorDB.query({
  vector: queryVector,
  topK: 6,
  includeMetadata: true
});

At this point you have raw text chunks ranked by relevance — but no answer yet. This is where generation comes in.

Step 4: Generate the Answer with Claude

Pass the retrieved chunks and the user's question to Claude as context, and ask it to answer strictly from the provided material. This keeps responses grounded and reduces hallucination.

If you're already using SubToAPI to turn your Claude access into an API, the call looks like this:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "max_tokens": 500,
    "messages": [
      {
        "role": "user",
        "content": "Using only the context below, answer the question. Cite the source title for each claim.\n\nContext:\n[CHUNK_1]\n[CHUNK_2]\n[CHUNK_3]\n\nQuestion: How do I reset a team member'\''s password?"
      }
    ]
  }'

The system prompt matters a lot here. Be explicit that the model should say "I don't know" if the context doesn't contain the answer — this is the single biggest lever against hallucinated responses in a knowledge base tool.

You are a documentation assistant. Answer only using the provided context.
If the answer isn't in the context, say you don't have that information.
Always cite the source document title in your answer.

For production use, stream the response so the UI feels responsive rather than waiting several seconds for a full answer. See /docs/streaming for details on consuming server-sent events from the API.

Step 5: Surface Citations and Fallbacks

Users trust a knowledge base tool more when they can verify the answer. Return the source titles and URLs alongside the generated text, and link each citation back to the metadata stored in Step 1.

Also build a fallback path: if the top retrieval score is below a confidence threshold, skip generation entirely and show "No relevant articles found" plus a link to contact support. This avoids confidently wrong answers on queries your knowledge base simply doesn't cover.

Step 6: Wire It Into Your Stack

Once retrieval and generation work, the remaining piece is exposing it as an API endpoint your frontend, Slack bot, or support widget can call. If you're prototyping and don't want to manage API key scoping and rate limits yourself, SubToAPI gives you a sub_live_... application key with streaming and usage metadata out of the box — see /docs/quickstart to get a working request in under five minutes, and /docs/messages for the full request/response reference.

Keeping It Accurate Over Time

A knowledge base search tool degrades if the underlying docs change and your index doesn't. Set up:

• A nightly or webhook-triggered re-ingestion job for changed documents.
• Chunk-level versioning so stale chunks get replaced, not duplicated.
• A feedback mechanism ("Was this helpful?") to flag chunks that consistently produce poor answers.

Questions

Do I need a vector database, or can I use regular SQL search? Regular full-text search works for exact keyword matches but misses semantically similar phrasing. For natural-language queries, a vector database (or pgvector extension on Postgres) gives much better retrieval accuracy.

How many chunks should I pass to the LLM per query? Five to eight relevant chunks is a reasonable default. More chunks increase the chance of covering the answer but also increase token cost and the risk of diluting relevance — tune based on your average document length.

Can I build this without managing my own Claude API key and infrastructure? Yes. SubToAPI converts your existing Claude access into an HTTPS API with streaming, tool use, and team seats, so you can focus on retrieval logic instead of key management — check /pricing for plan details or start a free trial at /signup.