How to Build a Semantic Search Engine with Claude

Semantic search finds results based on meaning rather than exact keyword matches. If someone searches "cancel a subscription" and your document says "terminate your plan," a keyword search misses it — a semantic search engine finds it because the two phrases are close in meaning. Claude is not an embedding model and doesn't generate vectors itself, so building a semantic search engine with Claude means combining a vector-based retrieval layer with Claude as the reasoning and synthesis layer on top.

This matters because most "semantic search" tutorials stop at vector similarity, which gets you close matches but not necessarily the right answer. Adding Claude on top lets you re-rank noisy retrieval results, filter out irrelevant chunks, and generate a direct answer instead of a list of snippets. This guide covers the full pipeline: embeddings, indexing, retrieval, and Claude-based re-ranking and synthesis.

The architecture at a glance

A production semantic search engine has four moving parts:

1. An embedding model that converts text into vectors (Claude does not do this — you need a dedicated embedding API or model, such as Voyage AI, OpenAI's embeddings, or an open-source model like bge-large).
2. A vector store (Pinecone, Weaviate, pgvector, or even an in-memory index for small datasets) that stores vectors and does nearest-neighbor search.
3. A retrieval step that takes a user query, embeds it, and pulls back the top N candidate chunks.
4. Claude, which takes those candidates, re-ranks them for actual relevance, and either returns the best matches or synthesizes a direct answer from them.

Skipping step 4 is why a lot of "semantic search" implementations feel sloppy — vector similarity alone often returns chunks that are topically close but don't actually answer the question.

Step 1: Embed and index your documents

Chunk your source documents into reasonably sized pieces (300–800 tokens works well for most content) and embed each chunk:

curl https://api.voyageai.com/v1/embeddings \
  -H "Authorization: Bearer $VOYAGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": ["Your subscription can be cancelled from account settings."],
    "model": "voyage-3"
  }'

Store the returned vector alongside the chunk text and a reference to the source document in your vector store. Keep the original text — you'll need it later to feed into Claude.

Step 2: Retrieve candidates for a query

When a search request comes in, embed the query with the same model and run a nearest-neighbor lookup:

const queryEmbedding = await getEmbedding(userQuery);
const candidates = await vectorStore.query({
  vector: queryEmbedding,
  topK: 15,
  includeMetadata: true,
});

Pull more candidates than you think you need (10–20) because vector similarity alone is noisy. The re-ranking step will narrow this down.

Step 3: Use Claude to re-rank and filter

This is where Claude earns its place in the pipeline. Pass the query and the candidate chunks to Claude and ask it to score relevance and discard anything that doesn't actually answer the question:

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    max_tokens: 800,
    messages: [{
      role: "user",
      content: `Query: "${userQuery}"

Rank the following passages by how directly they answer the query. Return a JSON array of indices in order of relevance, excluding passages that don't address the query at all.

${candidates.map((c, i) => `[${i}] ${c.text}`).join("\n\n")}`
    }],
  }),
});

Claude's output gives you a clean, relevance-ordered subset instead of a raw similarity list. This single step is usually where search quality jumps the most — it's cheap to add and catches cases where the embedding model returned something topically adjacent but not actually useful.

Step 4: Synthesize a direct answer

Once you have the top-ranked chunks, you can either return them as search results or go further and have Claude write a direct answer with citations:

const answer = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    max_tokens: 500,
    stream: true,
    messages: [{
      role: "user",
      content: `Answer the query using only the passages below. Cite which passage each claim comes from.

Query: ${userQuery}

Passages:
${topChunks.join("\n\n")}`
    }],
  }),
});

Streaming the response matters here — users searching expect fast feedback, not a 3-second wait for a full answer. See /docs/streaming for handling server-sent events on the client side.

When to add tool use

If your search engine needs to decide whether to hit the vector store at all, or whether to also query a structured database (prices, inventory, dates), define retrieval as a tool and let Claude decide when to call it:

{
  "name": "search_documents",
  "description": "Search the knowledge base for relevant passages",
  "input_schema": {
    "type": "object",
    "properties": {
      "query": { "type": "string" }
    },
    "required": ["query"]
  }
}

This turns your semantic search engine into an agent that can chain multiple retrievals for complex, multi-part questions. Details on schema and multi-turn tool calls are in /docs/tools.

Production concerns

Once this works in a prototype, the operational questions show up fast: who owns the API key, how do you track cost per query, and how do you avoid one runaway script burning your whole monthly quota. If you're running this inside a team, SubToAPI gives each service or developer its own scoped sub_live_ key against your Claude access, with per-key usage metadata so you can see exactly how much the re-ranking step versus the synthesis step is costing you. Plans start at the Solo tier (€9) for individual projects, with Team (€19/seat) and Scale (€49/seat) tiers adding seat management for larger search deployments — see /pricing for details, or start with the free trial at /signup.

The /docs/quickstart and /docs/messages pages cover the request format if you're integrating the Claude calls above into an existing backend.

questions

Does Claude generate embeddings for semantic search? No. Claude is a conversational/reasoning model, not an embedding model. You need a separate embedding model (Voyage AI, OpenAI embeddings, or an open-source alternative) to generate the vectors used for similarity search. Claude's role is re-ranking retrieved results and synthesizing answers.

How many candidate chunks should I retrieve before re-ranking? Retrieve 10–20 candidates from your vector store, then let Claude re-rank and filter down to the 3–5 most relevant. Retrieving too few risks missing the right answer; too many wastes tokens and slows the re-ranking step.

Can I skip the vector database for a small document set? Yes. If you have under a few thousand chunks, an in-memory cosine similarity search over stored vectors is fast enough and avoids the operational overhead of running Pinecone or Weaviate. Switch to a dedicated vector store once your corpus grows past that or you need filtering by metadata.