How to Build a RAG Pipeline with Claude

Retrieval-augmented generation (RAG) lets Claude answer questions using your own documents instead of relying only on what it learned during training. To build a RAG pipeline with Claude, you need four pieces: a way to split and embed your documents, a vector store to search them, a retrieval step that pulls the most relevant chunks for a given query, and a final call to Claude that injects those chunks into the prompt as context.

This article walks through each step with working code, so you end up with a pipeline you can run today and extend later — whether you're building internal docs search, a support bot, or a research assistant.

Why RAG instead of just a bigger context window

Claude's context windows are large, but stuffing entire document sets into every request is slow, expensive, and prone to burying the relevant information in noise. RAG solves this by retrieving only the chunks that matter for a specific query, keeping prompts small and answers grounded in your actual source material. It also lets you update your knowledge base without retraining or re-prompting from scratch — you just re-index.

Step 1: Chunk your documents

Split documents into chunks of roughly 300–800 tokens with some overlap (50–100 tokens) so context isn't lost at boundaries. Simple sentence or paragraph-based splitting works fine for most use cases; you don't need anything fancier to start.

function chunkText(text, maxChars = 1500, overlap = 200) {
  const chunks = [];
  let start = 0;
  while (start < text.length) {
    const end = Math.min(start + maxChars, text.length);
    chunks.push(text.slice(start, end));
    start += maxChars - overlap;
  }
  return chunks;
}

For structured docs (PDFs, Markdown, HTML), split on headings first, then apply this inside each section so you don't mix unrelated topics in one chunk.

Step 2: Generate embeddings and store them

Embed each chunk with an embeddings model (OpenAI's text-embedding-3-small, Voyage AI, or Cohere all work well with Claude) and store the vectors in a vector database — Pinecone, Qdrant, Weaviate, or even a local pgvector table for smaller datasets.

import { Pinecone } from '@pinecone-database/pinecone';

const pc = new Pinecone({ apiKey: process.env.PINECONE_API_KEY });
const index = pc.index('docs');

async function embedAndStore(chunks, embedFn) {
  const vectors = await Promise.all(
    chunks.map(async (text, i) => ({
      id: `chunk-${i}`,
      values: await embedFn(text),
      metadata: { text },
    }))
  );
  await index.upsert(vectors);
}

Claude itself doesn't generate embeddings — it's a generation model, not an embedding model — so you'll pair it with a dedicated embeddings provider for this step.

Step 3: Retrieve relevant chunks

At query time, embed the user's question with the same embedding model, then query your vector store for the top-k most similar chunks (k=3 to 6 is a reasonable starting point).

async function retrieve(question, embedFn, topK = 5) {
  const queryVector = await embedFn(question);
  const results = await index.query({
    vector: queryVector,
    topK,
    includeMetadata: true,
  });
  return results.matches.map(m => m.metadata.text);
}

If retrieval quality is inconsistent, add a reranking step (e.g. Cohere Rerank) on top of the initial vector search before passing chunks to Claude — this often improves answer accuracy more than tuning chunk size.

Step 4: Build the prompt and call Claude

Inject the retrieved chunks into the system prompt or as part of the user message, clearly labeled, and instruct Claude to answer only from the provided context (or to say it doesn't know if the context is insufficient). This is the step where routing through SubToAPI is useful: you get a single sub_live_... API key, streaming support, and usage metadata per request without managing provider-specific auth separately.

async function askWithContext(question, chunks) {
  const context = chunks.map((c, i) => `[${i + 1}] ${c}`).join('\n\n');

  const response = await fetch('https://api.subtoapi.app/v1/messages', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${process.env.SUBTOAPI_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'claude-sonnet-4-5',
      max_tokens: 1024,
      system: `Answer the user's question using only the context below. 
If the context doesn't contain the answer, say you don't know.

Context:
${context}`,
      messages: [{ role: 'user', content: question }],
    }),
  });

  const data = await response.json();
  return data.content[0].text;
}

Full request and response shapes are documented at /docs/messages, and streaming responses for longer answers are covered at /docs/streaming.

Putting it together

async function ragQuery(question) {
  const chunks = await retrieve(question, embedFn);
  return askWithContext(question, chunks);
}

This is the full loop: chunk once at ingestion time, embed and store, then retrieve and generate on every query. Everything downstream of retrieval — prompt construction, model call, response handling — is where most of your iteration time will go. Start with a small, high-quality corpus and tune chunk size, top-k, and your system prompt before scaling up the dataset.

Common pitfalls

• Chunks too large or too small: too large dilutes relevance, too small loses context. Test with real queries, not guesses.
• No citation or source tracking: store chunk metadata (source file, page number) so Claude's answers can reference where information came from.
• Ignoring retrieval failures: if no chunk is relevant, don't force Claude to answer — instruct it to say so explicitly in the system prompt.
• Mixing embedding models: always embed queries and documents with the same model and dimensionality, or similarity scores become meaningless.

If you're running this in production, check /pricing for plan details — Solo works for prototypes and side projects, while Team and Scale add seats and higher limits for multi-developer setups. A free trial is available at /signup, and the quickstart at /docs/quickstart covers getting your first key and making a request in under five minutes.

Questions

Does Claude generate its own embeddings? No. Claude is a generation model. Pair it with a separate embeddings provider (OpenAI, Voyage, Cohere) for the retrieval half of the pipeline.

How many chunks should I retrieve per query? Start with 3–6 chunks (top-k). Fewer chunks keep prompts focused; more chunks help on complex, multi-part questions but increase token cost and noise.

Can I use Claude's tool use instead of manual retrieval? Yes — you can expose a retrieval function as a tool Claude calls dynamically rather than always injecting context upfront. See /docs/tools for how tool use works with SubToAPI.