Claude API for Internal Knowledge Search Tools

Internal knowledge search tools answer a specific question: "how do I find the right answer buried across our wikis, tickets, and docs without asking a human?" The Claude API is well suited to this because it can read large amounts of retrieved text, reason across inconsistent sources, and produce a direct answer with citations instead of a list of links. This article covers the architecture pattern that works in production, where Claude fits in the pipeline, and how to wire it up with tool use and streaming.

The short answer: you don't ask Claude to "know" your internal docs from training data — you retrieve the relevant chunks yourself (search, vector search, or a hybrid), then hand them to Claude as context and let it synthesize the answer. This is the retrieval-augmented generation (RAG) pattern, and for internal knowledge search it's almost always the right one because your content changes constantly and you need accurate, current answers, not memorized ones.

Why Claude fits internal knowledge search

Three properties matter here:

• Large context windows let you pass in multiple full documents, not just snippets, so Claude can reconcile conflicting or overlapping information.
• Tool use lets Claude decide when to call your internal search function instead of answering blind, and it can issue follow-up searches if the first retrieval wasn't enough.
• Instruction following means you can force citation formats, refuse to answer when no source supports a claim, and keep tone consistent with internal documentation style.

This combination is what separates a useful internal search assistant from a chatbot that confidently makes things up.

The architecture

A typical internal knowledge search tool has four parts:

1. Indexer — chunks your docs (wiki pages, PDFs, tickets, Slack exports) and stores embeddings in a vector store or search index.
2. Retriever — takes a user query, searches the index, returns the top N chunks with source metadata.
3. Claude call — receives the query plus retrieved chunks, generates an answer with inline citations.
4. Access layer — filters what each user is allowed to retrieve before anything reaches Claude.

The access layer matters more for internal tools than for public-facing ones. If your HR docs, security runbooks, and engineering wiki are all in one index, you need to filter by the requesting user's permissions before retrieval, not after — otherwise Claude might cite something the user shouldn't see.

Basic retrieval + generation call

const retrievedChunks = await searchInternalDocs(userQuery, { userId });

const contextBlock = retrievedChunks
  .map((c, i) => `[Source ${i + 1}: ${c.title}]\n${c.text}`)
  .join("\n\n");

const response = await fetch("https://api.anthropic.com/v1/messages", {
  method: "POST",
  headers: {
    "x-api-key": process.env.ANTHROPIC_API_KEY,
    "anthropic-version": "2023-06-01",
    "content-type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-opus-4",
    max_tokens: 1024,
    system:
      "You are an internal search assistant. Answer only using the provided sources. Cite sources as [Source N]. If the sources don't contain the answer, say so explicitly.",
    messages: [
      {
        role: "user",
        content: `Sources:\n${contextBlock}\n\nQuestion: ${userQuery}`,
      },
    ],
  }),
});

This is the minimum viable version. It works well for single-shot Q&A, but real internal search often needs Claude to run multiple searches — for example, a vague question like "what's our policy on expense approvals over $500" might need a second, more targeted query.

Letting Claude drive retrieval with tool use

Instead of always doing retrieval yourself before the call, you can define a search function as a tool and let Claude decide when and how to call it. This is useful when the first retrieval is ambiguous or the question needs decomposition into sub-queries.

{
  "name": "search_internal_docs",
  "description": "Search the internal knowledge base for relevant documents.",
  "input_schema": {
    "type": "object",
    "properties": {
      "query": { "type": "string" },
      "max_results": { "type": "integer" }
    },
    "required": ["query"]
  }
}

Claude will return a tool_use block when it decides it needs more information, your backend executes the search against the user's permitted index, and you send the results back as a tool_result in the next message. This loop can repeat a few times before Claude gives a final answer. If you're new to this pattern, the mechanics of defining tools and handling the tool_use/tool_result cycle are covered in detail at /docs/tools.

Streaming for a responsive UI

Internal tools get used constantly throughout the day, so response latency matters. Streaming the answer token-by-token instead of waiting for the full completion makes the tool feel instant even on longer answers. If you're building the frontend yourself, /docs/streaming walks through handling server-sent events for this.

Where SubToAPI fits

If you already pay for a Claude subscription and want to expose it as a proper API for an internal tool — with per-team API keys, usage metadata per request, and streaming support — SubToAPI turns that subscription into an HTTPS endpoint without a separate Anthropic billing setup. You generate a sub_live_... key per team or environment, point your retrieval backend at it, and get the same Messages-style interface.

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-4",
    "max_tokens": 1024,
    "messages": [
      {"role": "user", "content": "Sources:\n[Source 1] ...\n\nQuestion: What is our expense approval limit?"}
    ]
  }'

This is handy for internal tools specifically because you often want separate keys per department or environment (staging vs. production search index) without juggling multiple Anthropic accounts. Plans start at Solo €9 for a single key, with Team at €19/seat and Scale at €49/seat for larger rollouts — see /pricing for the breakdown, or start with a free trial at /signup. The /docs/quickstart and /docs/messages pages cover the request format if you're migrating an existing integration.

Practical tips for accuracy

• Always instruct Claude to say "I don't know" or "not found in sources" rather than guessing — this single system prompt line eliminates most hallucination complaints.
• Include document titles and update dates in the context so Claude can prefer newer sources when documents conflict.
• Log every query and the chunks retrieved for it — this is your fastest way to debug "why did it say that" complaints from users.
• Keep the number of retrieved chunks reasonable (5–10 is typical); dumping 50 chunks into context increases cost and often hurts answer focus.

Do I need a vector database to build this?

Not strictly. Keyword or hybrid search works fine for smaller knowledge bases (a few thousand documents). Vector search helps most when queries use different wording than the source documents.

Can Claude search documents it wasn't trained on?

Yes, as long as you retrieve and pass the relevant text in the request. Claude has no automatic access to your internal systems — retrieval is your responsibility, generation is Claude's.

How do I prevent Claude from answering outside the provided sources?

Use an explicit system prompt instruction and test it with known "no answer available" queries. Review logs periodically to catch cases where it answers from general knowledge instead of the sources you gave it.