How to Build a Claude-Powered Data Extraction Pipeline

What a Claude-Powered Data Extraction Pipeline Actually Is

A data extraction pipeline built on Claude takes unstructured input — PDFs, emails, scraped HTML, scanned documents, support tickets — and turns it into structured, machine-readable output (usually JSON) that your database, CRM, or analytics tool can consume directly. The core idea is simple: instead of writing brittle regex or custom parsers for every document format, you describe the schema you want and let Claude read the content and fill it in.

The hard part isn't getting Claude to extract data once — that's a five-line prompt. The hard part is making it reliable at scale: consistent JSON shape, predictable error handling, retries, batching, and cost control across thousands of documents a day. This article covers the architecture decisions that make the difference between a demo and a production pipeline.

Step 1: Define a Strict Output Schema

Before writing any prompt, define the exact shape of the data you want back. Don't leave this to Claude's judgement — ambiguity is where extraction pipelines break.

{
  "invoice_number": "string",
  "vendor_name": "string",
  "total_amount": "number",
  "currency": "string",
  "line_items": [
    { "description": "string", "quantity": "number", "unit_price": "number" }
  ],
  "due_date": "string | null"
}

Give Claude this schema explicitly in the system prompt, and tell it to return null for missing fields rather than guessing. Models hallucinate less when "I don't know" is an explicitly allowed answer.

Step 2: Write an Extraction Prompt That Enforces Structure

A good extraction system prompt does three things: constrains the output format, defines field semantics, and tells the model what to do with ambiguous or missing data.

You are a data extraction engine. Extract the fields defined in the schema below from the provided document.

Rules:
- Return valid JSON only, no commentary, no markdown fences.
- If a field is not present in the document, return null.
- Dates must be in ISO 8601 format (YYYY-MM-DD).
- Numbers must not include currency symbols or thousands separators.

Schema:
{schema here}

Put the document content in the user message, not the system prompt — this keeps your system prompt reusable across every document you process and makes caching more effective if your provider supports it.

Step 3: Call the API and Parse the Response

Here's a minimal extraction call. If you're using SubToAPI to turn your Claude access into a standard HTTPS API, the request looks like this:

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet-4-5",
    max_tokens: 1024,
    system: extractionSystemPrompt,
    messages: [{ role: "user", content: documentText }]
  })
});

const data = await response.json();
const extracted = JSON.parse(data.content[0].text);

Wrap the JSON.parse in a try/catch. Even with strict instructions, models occasionally wrap output in explanation text or markdown fences. Strip common wrappers before parsing:

function parseExtraction(raw) {
  const cleaned = raw.replace(/```json|```/g, "").trim();
  return JSON.parse(cleaned);
}

Step 4: Validate Before You Trust It

Never pipe extracted JSON straight into your database. Validate against a schema library (Zod, Yup, or JSON Schema) so malformed or partial extractions fail loudly instead of silently corrupting your data:

const InvoiceSchema = z.object({
  invoice_number: z.string(),
  vendor_name: z.string(),
  total_amount: z.number(),
  currency: z.string(),
  line_items: z.array(z.object({
    description: z.string(),
    quantity: z.number(),
    unit_price: z.number()
  })),
  due_date: z.string().nullable()
});

const result = InvoiceSchema.safeParse(extracted);
if (!result.success) {
  // route to a dead-letter queue for manual review
}

This validation layer is what separates a prototype from something you'd run in production. Documents that fail validation should go to a review queue, not get silently dropped.

Step 5: Batch, Rate-Limit, and Retry

Real pipelines process documents in bulk, which means you need to handle concurrency and transient failures without flooding the API.

• Concurrency limits: cap parallel requests (5–10 is a reasonable starting point) to avoid rate-limit errors.
• Exponential backoff: retry on 429s and 5xxs with increasing delay, not fixed intervals.
• Idempotency: track which document IDs have already been processed so retries don't duplicate work.
• Dead-letter handling: after N retries, log the failure and move on rather than blocking the whole batch.

async function extractWithRetry(doc, attempt = 1) {
  try {
    return await extractDocument(doc);
  } catch (err) {
    if (attempt >= 4) throw err;
    await sleep(2 ** attempt * 500);
    return extractWithRetry(doc, attempt + 1);
  }
}

Step 6: Track Usage and Cost Per Document

Extraction pipelines process large volumes, so token cost adds up fast. Log token usage per request so you can spot documents that are unusually expensive to process — often a sign of noisy OCR output or a document that's too long and should be chunked first. If you're running the pipeline through SubToAPI, usage metadata comes back with every response, so you can attribute cost per document, per customer, or per pipeline stage without building your own tracking layer. See /docs/messages for the full response shape.

Step 7: Handle Long Documents With Chunking

For documents that exceed a comfortable context size — long contracts, multi-page reports — split by logical boundaries (sections, pages) rather than fixed character counts, extract each chunk separately, then merge results. For fields that span chunks (like totals), run a final pass over the merged partial results to reconcile them.

Should You Use Streaming for Extraction?

Usually not. Extraction output is JSON you need in full before parsing, so streaming adds complexity without benefit for this use case — reserve streaming (see /docs/streaming) for user-facing text generation, not structured extraction jobs.

Getting Started Quickly

If you already have Claude access and want to skip building your own API wrapper, retry logic, and usage tracking from scratch, SubToAPI turns your existing Claude subscription into a standard HTTPS API with application keys, built-in usage metadata, and team seats. Start with the /docs/quickstart guide, or sign up at /signup to get a sub_live_ key and run your first extraction call in minutes. Plans start at €9/month — see /pricing for details.

Questions

Do I need function calling / tools for data extraction, or is a plain prompt enough? A well-structured prompt with a strict JSON schema is enough for most extraction tasks. Tool use (see /docs/tools) becomes useful when extraction needs to trigger follow-up actions, like looking up a vendor ID or validating a field against an external system mid-extraction.

How do I handle documents with inconsistent formatting, like scanned invoices from different vendors? Keep the schema fixed but make the prompt format-agnostic — describe field semantics ("the total amount owed," not "the number next to 'Total'"), and always allow null. This lets one prompt handle structural variation across vendors without a template per source.

What's the biggest cause of extraction pipelines breaking in production? Trusting model output without validation. Pipelines that parse JSON and write straight to a database break silently when a field is missing or malformed. Add a validation layer and a dead-letter queue from day one, not after the first bad write.