Claude API for Internal Knowledge Base Search

If you're searching for how to use the Claude API for internal knowledge base search, you're likely trying to let employees ask questions in plain English and get answers pulled from your company's docs, wikis, tickets, or Slack history — instead of making them grep through Confluence or Notion. Claude is well suited to this because it's strong at reading long, messy documents and synthesizing an answer, not just returning a list of matching pages like a keyword search engine would.

There are two practical architectures for this: retrieval-augmented generation (RAG), where you fetch relevant chunks from a vector store and hand them to Claude, and long-context stuffing, where you pass entire documents (or a curated subset) directly in the prompt and let Claude do the searching itself. Most production systems end up using a hybrid: retrieval to narrow down candidates, then Claude to reason over the top results and write a grounded answer with citations.

RAG vs Long Context: Which to Use

RAG makes sense when your knowledge base is large — thousands of documents, changelogs, support tickets — and you can't fit everything into a single prompt. You embed your documents, store vectors in something like pgvector, Pinecone, or Weaviate, retrieve the top-k chunks for a query, and pass those chunks to Claude along with the user's question.

Long context makes sense for smaller, well-scoped knowledge bases — a single product's documentation, an onboarding handbook, a set of design docs — where you can fit the relevant material directly into the prompt without retrieval infrastructure. Claude models support large context windows, so "just paste the docs in" is a legitimate strategy for teams that don't want to run a vector database.

Many internal search tools start with long-context stuffing because it's faster to ship, then move to RAG once the corpus outgrows what fits comfortably in a single request.

Building a Simple RAG Pipeline

The core loop is: chunk your documents, embed the chunks, retrieve by similarity, then call Claude with the retrieved context and a clear instruction to only answer from what's provided.

A minimal request to Claude after retrieval looks like this:

curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 1024,
    "system": "You are an internal search assistant. Answer only using the provided context. If the answer is not in the context, say you cannot find it, and cite the source document for every claim.",
    "messages": [
      {
        "role": "user",
        "content": "Context:\n[doc: onboarding.md]\nNew hires get a laptop within 3 business days...\n\nQuestion: How long until a new hire gets a laptop?"
      }
    ]
  }'

Keep the system prompt strict about grounding — explicitly telling Claude to say "I don't know" when the retrieved chunks don't contain the answer is the single most effective way to cut down on hallucinated answers in internal search tools.

If you're routing this traffic through SubToAPI instead of calling Anthropic directly, the same request goes through the /v1/messages endpoint with your sub_live_... key, and you get usage metadata per request out of the box — useful for tracking which teams or apps are querying the knowledge base most heavily:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 1024,
    "messages": [{ "role": "user", "content": "How long until a new hire gets a laptop?" }]
  }'

See the messages docs for the full request shape.

Tool Calling for Live Search

Instead of retrieving once and stuffing context up front, you can give Claude a search tool and let it decide when and what to query. This works well when the knowledge base is large and the right query terms aren't obvious from the user's question alone — Claude can issue multiple searches, refine terms, and combine results before answering.

{
  "tools": [
    {
      "name": "search_knowledge_base",
      "description": "Search internal docs and return matching passages",
      "input_schema": {
        "type": "object",
        "properties": {
          "query": { "type": "string" }
        },
        "required": ["query"]
      }
    }
  ]
}

Claude will call search_knowledge_base with a query, you run it against your search index, return the results as a tool result message, and Claude uses them to compose the final answer — optionally calling the tool again if the first results aren't sufficient. This pattern is worth using when questions are ambiguous ("what's our policy on X") and a single fixed retrieval step tends to miss relevant documents. See tool use for the message flow.

Streaming Answers for a Better UX

Internal search tools feel much more responsive when answers stream token-by-token instead of appearing all at once after a multi-second wait — especially once you add retrieval latency on top of generation time. Server-sent event streaming works the same way whether you're calling Anthropic directly or through SubToAPI; see streaming for the setup.

Team Access and Key Management

Internal knowledge base search is usually a shared tool, not a single developer's side project — support, sales, and engineering all end up querying the same system. That means you need per-application keys, usage visibility, and a way to add teammates without sharing one raw API key in a shared .env file.

This is where SubToAPI's dashboard is useful in practice: you generate scoped sub_live_... keys per application (internal search bot, Slack integration, support tool), see usage per key, and add team seats without managing individual Anthropic billing. Plans start at €9/month for a solo builder and scale to €19–€49/seat for teams running multiple internal tools against the same Claude access. Check pricing or start with the quickstart if you're setting this up for the first time.

Security Considerations

Internal knowledge bases often contain sensitive material — HR policies, financial figures, customer data. A few practical rules:

• Scope retrieval to what the requesting user is actually allowed to see; don't rely on the LLM to enforce access control.
• Redact or exclude highly sensitive documents from the searchable index entirely rather than trusting prompt instructions to withhold them.
• Log queries and retrieved sources so you can audit what was surfaced, especially early on while you're tuning retrieval quality.
• Keep API keys per-application rather than one shared key across every internal tool, so you can revoke access to a single integration without breaking everything else.

FAQ

Do I need a vector database to use Claude for internal search? No. If your knowledge base is small enough to fit in the context window, you can paste relevant documents directly into the prompt. A vector database becomes worthwhile once your corpus is too large to include in full and you need retrieval to narrow it down first.

How do I stop Claude from making up answers not in my docs? Give it an explicit system instruction to only answer from the provided context and to say when it can't find something, and always include source citations in the retrieved chunks so answers can be verified against the original document.

Should I use RAG or tool calling for search? Use RAG when a single retrieval step reliably surfaces the right documents. Use tool calling when queries are ambiguous and Claude benefits from issuing multiple, refined searches before answering — it costs more round trips but improves accuracy on harder questions.