Claude API RAG Pipeline Integration Guide

Retrieval-augmented generation (RAG) combines a search step with a language model call: you retrieve relevant chunks of your own data, inject them into the prompt, and let Claude generate an answer grounded in that content. Integrating Claude into a RAG pipeline means solving four problems — chunking your source data, storing and querying embeddings, assembling a prompt that keeps Claude grounded, and handling the API call itself (including streaming and errors) in production.

This guide walks through each stage with working code, then covers the practical details that trip people up: context window budgeting, citation formatting, and how to avoid hallucinated answers when retrieval returns weak matches. If you're already calling the Claude API directly, you can swap in SubToAPI at the final step without changing anything else in your pipeline — the retrieval and chunking logic stays identical.

The RAG pipeline at a glance

A production RAG system has five moving parts:

1. Ingestion — split documents into chunks small enough to embed and retrieve precisely.
2. Embedding — convert each chunk into a vector using an embedding model.
3. Storage — index vectors in a vector database (pgvector, Pinecone, Weaviate, Qdrant, or even an in-memory store for small datasets).
4. Retrieval — at query time, embed the user's question and fetch the top-k most similar chunks.
5. Generation — build a prompt with the retrieved chunks and call Claude to produce the final answer.

Claude's role is strictly the last step. It doesn't do retrieval for you — that's your embedding/vector store layer — but it does need a prompt structure that makes good use of what you retrieve.

Step 1: Chunk your documents sensibly

Chunk size matters more than most teams expect. Too large, and you waste context window and dilute relevance. Too small, and you lose surrounding context the model needs to answer correctly.

A reasonable default: 300–500 tokens per chunk, with 10–15% overlap between consecutive chunks so you don't cut a sentence or table row in half.

function chunkText(text, maxTokens = 400, overlapTokens = 50) {
  const words = text.split(/\s+/);
  const wordsPerChunk = maxTokens * 0.75; // rough token-to-word ratio
  const overlapWords = overlapTokens * 0.75;
  const chunks = [];

  for (let i = 0; i < words.length; i += wordsPerChunk - overlapWords) {
    chunks.push(words.slice(i, i + wordsPerChunk).join(" "));
  }
  return chunks;
}

For structured content (API docs, legal contracts, support tickets), chunk by natural boundaries — headings, sections, Q&A pairs — rather than fixed word counts. It almost always improves retrieval precision.

Step 2: Embed and store

Use any embedding model your vector store supports. The embedding provider doesn't need to match Claude — plenty of teams embed with one model and generate with Claude.

import { Pool } from "pg";
const db = new Pool();

async function storeChunk(text, embedding, metadata) {
  await db.query(
    `INSERT INTO documents (content, embedding, metadata) VALUES ($1, $2, $3)`,
    [text, embedding, metadata]
  );
}

At query time, embed the user's question and run a similarity search:

async function retrieveChunks(queryEmbedding, k = 5) {
  const result = await db.query(
    `SELECT content, metadata FROM documents
     ORDER BY embedding <=> $1 LIMIT $2`,
    [queryEmbedding, k]
  );
  return result.rows;
}

Step 3: Build a grounded prompt

This is where most RAG integrations go wrong. Dump raw chunks into the system prompt without structure, and Claude either ignores weak matches or blends them incoherently. Instead, label each chunk and instruct Claude explicitly on how to use them.

function buildPrompt(question, chunks) {
  const context = chunks
    .map((c, i) => `[Source ${i + 1}]\n${c.content}`)
    .join("\n\n");

  return {
    system: `You are a support assistant. Answer using only the sources provided below.
If the sources don't contain enough information to answer, say so explicitly —
do not guess. Cite sources by number in brackets, e.g. [Source 2].

${context}`,
    user: question,
  };
}

Explicitly telling Claude it's allowed to say "I don't know" is the single most effective guardrail against hallucinated answers in a RAG system.

Step 4: Call Claude and stream the response

Once the prompt is assembled, the generation call itself is a standard Messages API request. If you're routing through SubToAPI, the request shape is the same — you're just hitting a single HTTPS endpoint with an application key instead of managing provider credentials directly:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 1024,
    "system": "You are a support assistant...",
    "messages": [
      { "role": "user", "content": "How do I reset my API key?" }
    ],
    "stream": true
  }'

Streaming matters for RAG specifically because generation often starts after a noticeable retrieval delay (embedding the query, running the vector search). Streaming the model's response keeps the perceived latency low even if the retrieval step added 200–400ms upfront. See the streaming docs for the SSE event format.

For teams running RAG across multiple products or customers, SubToAPI's per-key usage metadata is useful here — each application key maps to a product or tenant, so you can see token consumption per RAG deployment without building your own tracking layer. Check pricing for Solo, Team, and Scale tiers.

Budgeting the context window

Claude's context window is large, but "large" isn't an excuse to skip retrieval filtering. Stuffing 20 chunks into every request increases cost and can actually reduce answer quality — the model has to work harder to find the relevant needle. Keep top-k between 3 and 8 chunks for most use cases, and re-rank retrieved chunks with a cheaper model or a cross-encoder before sending them to Claude if your corpus is large and noisy.

Handling tool-augmented retrieval

If your retrieval step itself should be dynamic — for example, letting Claude decide whether to search a knowledge base, query a database, or answer from general knowledge — define retrieval as a tool rather than a fixed pre-step. See the tool use docs for the function-calling schema; the pipeline becomes: Claude decides to call search_knowledge_base, your backend runs the vector query, and the results are returned as a tool result for Claude to generate from.

Getting started

If you want to prototype this quickly: sign up for a free trial, get a sub_live_... key, and follow the quickstart to send your first grounded message. The Messages API reference covers the full request schema if you need multi-turn conversation history alongside retrieved context.

Questions

Does Claude do the retrieval itself, or do I need a separate vector database? Claude only generates text — it has no built-in retrieval. You need an embedding model and a vector store (pgvector, Pinecone, Qdrant, etc.) to find relevant chunks before calling Claude.

How many retrieved chunks should I send per request? Start with 3–8 chunks of 300–500 tokens each. More isn't always better — oversized context dilutes relevance and increases cost without improving answer quality.

Can I use RAG with tool use instead of a fixed retrieval step? Yes. Define retrieval as a tool so Claude decides when to search versus answer directly, which works better for mixed knowledge-base and general-knowledge queries — see /docs/tools.