Claude API RAG Pipeline Implementation Tutorial

A retrieval-augmented generation (RAG) pipeline lets Claude answer questions using your own documents instead of relying only on what it learned during training. This guide walks through the full implementation: chunking your content, generating embeddings, storing and querying a vector index, and building the final prompt that gets sent to the Claude API.

By the end you'll have a working pipeline you can run against a knowledge base, a support docs folder, or any corpus of text, with Claude generating grounded answers and (optionally) citing the source chunks it used.

What a RAG pipeline actually does

RAG solves a specific problem: large language models have a fixed knowledge cutoff and no access to your private data. Instead of fine-tuning a model (slow, expensive, hard to update), you retrieve relevant text at query time and inject it into the prompt. Claude then reasons over that text to produce an answer.

The pipeline has four moving parts:

1. Ingestion — split documents into chunks and embed them
2. Storage — put embeddings in a vector database (or in-memory index for small corpora)
3. Retrieval — at query time, embed the user's question and find the closest chunks
4. Generation — send the retrieved chunks plus the question to Claude and return the answer

Step 1: Chunk your documents

Chunk size matters more than most people expect. Too large and you waste context window and dilute relevance; too small and you lose surrounding context the model needs to answer correctly.

A reasonable default: 300–500 tokens per chunk, with 10-15% overlap between consecutive chunks so you don't cut a sentence or idea in half.

function chunkText(text, chunkSize = 400, overlap = 50) {
  const words = text.split(/\s+/);
  const chunks = [];
  for (let i = 0; i < words.length; i += chunkSize - overlap) {
    chunks.push(words.slice(i, i + chunkSize).join(" "));
  }
  return chunks;
}

Store each chunk alongside metadata: source filename, section title, page number. You'll need this later for citations.

Step 2: Generate embeddings and store them

Embed each chunk using whatever embedding model you already have access to (OpenAI, Cohere, Voyage, or a local model all work — Claude doesn't need to be involved at this stage). Store the vectors in a vector database like Pinecone, Weaviate, Qdrant, or even an in-memory array with cosine similarity if your corpus is small (under a few thousand chunks).

import { embed } from "./embeddings.js";

const records = [];
for (const chunk of chunks) {
  const vector = await embed(chunk.text);
  records.push({ vector, text: chunk.text, metadata: chunk.metadata });
}
await vectorStore.upsert(records);

Batch your embedding calls where the provider allows it — embedding one chunk at a time on a large corpus is slow and often rate-limited unnecessarily.

Step 3: Retrieve relevant chunks at query time

When a user asks a question, embed the question with the same embedding model and query the vector store for the top-k most similar chunks (k=3 to 8 is typical).

const queryVector = await embed(userQuestion);
const results = await vectorStore.query(queryVector, { topK: 5 });
const context = results.map(r => r.text).join("\n\n---\n\n");

If your corpus is noisy, add a reranking step here: a lightweight model or even Claude itself can score the top-20 retrieved chunks and keep only the 5 most relevant before building the final prompt. This meaningfully improves answer quality for larger knowledge bases.

Step 4: Build the prompt and call Claude

This is where the retrieval and generation steps meet. Put the retrieved context in the system prompt or as a prefixed user message, be explicit that Claude should only answer from the provided context, and ask it to say so when the answer isn't present.

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-4",
    "max_tokens": 1024,
    "system": "Answer only using the context below. If the answer is not in the context, say you don'\''t know.\n\nCONTEXT:\n'"$CONTEXT"'",
    "messages": [
      { "role": "user", "content": "What is our refund policy for annual plans?" }
    ]
  }'

This is the step where most teams reach for SubToAPI rather than managing raw API credentials — you get a stable sub_live_... key, usage metadata per request, and a single dashboard for the whole team instead of juggling separate access setups. The Messages API shape is the same one documented at /docs/messages.

Step 5: Stream the response for better UX

If this pipeline backs a chat interface, stream tokens back to the client instead of waiting for the full response. This matters more in RAG than in plain chat because context injection adds latency before generation even starts — streaming hides that.

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-opus-4",
    max_tokens: 1024,
    stream: true,
    system: systemPromptWithContext,
    messages: [{ role: "user", content: userQuestion }]
  })
});

Full streaming setup, including how to parse server-sent events on the client, is covered at /docs/streaming.

Step 6: Add citations with tool use

For production RAG, users want to know where an answer came from. Use tool calling to force Claude to return structured citations alongside the answer — define a tool schema that requires a sources array referencing chunk IDs, and Claude will populate it instead of burying citations in free text. Details on defining and handling tools are at /docs/tools.

Production considerations

• Chunk overlap and metadata matter more than model choice for answer quality — fix retrieval before tuning prompts.
• Cache embeddings for unchanged documents; re-embedding on every deploy is wasted cost and time.
• Track token usage per query so you can see which questions are expensive (large context injections) and optimize chunk counts.
• Set a context budget — don't blindly stuff 20 chunks into the prompt; pick the smallest set that answers the question.
• Log retrieval hits vs. misses to find gaps in your corpus before users complain about wrong answers.

If you're prototyping this before committing to infrastructure, start with the free trial at /signup and the quickstart at /docs/quickstart — you can have the generation step working end-to-end in a few minutes, then spend your real engineering time on the retrieval quality, which is almost always the actual bottleneck.

questions

Do I need Claude to generate the embeddings too? No. Embeddings and generation are separate concerns. Use any embedding provider (OpenAI, Voyage, Cohere, or a local model) for retrieval, and use Claude only for the final reasoning and answer generation step.

How many chunks should I retrieve per query? Start with 5 and adjust based on answer quality. More chunks means more context but also more noise and higher token cost — add a reranking step if you need to retrieve 15-20 and narrow down before sending to Claude.

Can I run this pipeline without managing my own API keys and billing? Yes — a service like SubToAPI gives you a single application key, usage metadata per request, and streaming support, so you can focus on the retrieval logic instead of API plumbing.