Claude API Content Moderation Pipeline Guide

A content moderation pipeline built on the Claude API classifies user-generated text (and sometimes images) for policy violations, then routes flagged content to automated actions or human review. This guide covers the actual architecture: how to structure the classification prompt, get machine-readable output, set confidence thresholds, and avoid the two failure modes that break most moderation systems — over-blocking legitimate content and silently missing genuine violations.

If you're searching for how to wire Claude into a moderation workflow, the short answer is: don't ask Claude a yes/no question in free text. Ask it to return structured categories with severity scores, enforce that structure with tool use or JSON mode, and build your business logic around the scores rather than around parsing prose.

Why Claude works well for this

Generic keyword or regex filters miss context — sarcasm, reclaimed slurs, coded harassment, and borderline medical or violent content that's legitimate in one context and not in another. Claude handles this well because it reads the full message in context rather than matching patterns. The tradeoff is latency and cost per call, which is why the pipeline design below separates cheap pre-filtering from expensive LLM classification.

Pipeline architecture

A production moderation pipeline usually has three stages:

1. Pre-filter — cheap, deterministic checks (blocklists, regex, length limits, rate limits) that catch obvious cases without calling the API.
2. Classify — the content that passes pre-filtering goes to Claude for contextual classification across your policy categories.
3. Decide and escalate — your application logic maps Claude's output to an action: allow, soft-warn, block, or queue for human review.

This keeps API calls limited to ambiguous content instead of every single message, which matters once you're moderating at volume.

Designing the classification prompt

Give Claude a fixed taxonomy and ask it to score against it, not to make a single pass/fail call. A narrow, explicit taxonomy produces far more consistent results than an open-ended "is this inappropriate?" prompt.

You are a content moderation classifier. Score the message below against these
categories on a scale of 0-3 (0 = none, 1 = mild, 2 = moderate, 3 = severe):

- harassment
- hate_speech
- sexual_content
- violence
- self_harm
- spam

Return only valid JSON matching this shape:
{"harassment": 0, "hate_speech": 0, "sexual_content": 0, "violence": 0,
 "self_harm": 0, "spam": 0, "reasoning": "one short sentence"}

Message: "{{user_message}}"

The reasoning field is small but important — it gives you an audit trail when you need to explain a decision later, and it's cheap to generate alongside the scores.

Enforcing structured output

Free-text JSON is good enough for prototypes but breaks under load when the model adds commentary or formatting drift. Use tool use (function calling) to force a strict schema instead:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet",
    "max_tokens": 200,
    "tools": [{
      "name": "moderation_result",
      "description": "Return moderation scores for the message",
      "input_schema": {
        "type": "object",
        "properties": {
          "harassment": {"type": "integer"},
          "hate_speech": {"type": "integer"},
          "sexual_content": {"type": "integer"},
          "violence": {"type": "integer"},
          "self_harm": {"type": "integer"},
          "spam": {"type": "integer"},
          "reasoning": {"type": "string"}
        },
        "required": ["harassment","hate_speech","sexual_content","violence","self_harm","spam"]
      }
    }],
    "tool_choice": {"type": "tool", "name": "moderation_result"},
    "messages": [{"role": "user", "content": "Classify: {{user_message}}"}]
  }'

This guarantees a parseable tool call back instead of hoping the model stays in JSON. SubToAPI's /v1/messages endpoint speaks the same Messages API shape Claude uses natively, so if you've already built tool-based prompts, they drop in unchanged — see /docs/tools and /docs/messages for the full schema.

Setting thresholds and routing decisions

Map scores to actions in your own code, not in the prompt. A reasonable starting point:

• 0–1 across all categories → allow immediately
• 2 in any single category → allow but log, or apply a soft warning
• 3 in any category, or 2+ in two or more categories → block and queue for human review

Tune these per category. Spam at severity 2 might auto-block, while self-harm at severity 1 might still route to a human reviewer because false negatives there carry real cost. Don't treat all categories with the same threshold — that's the most common design mistake in early moderation pipelines.

Handling multi-turn and context

Single-message classification misses escalation patterns — a conversation that turns hostile over several turns, or coordinated harassment spread across messages. For conversational content, pass the last 3–5 turns as context instead of classifying each message in isolation:

const resp = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "content-type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet",
    max_tokens: 200,
    messages: [
      { role: "user", content: `Conversation context:\n${recentTurns}\n\nClassify the final message for moderation.` }
    ]
  })
});

This catches gradual escalation that per-message classification misses entirely.

Logging and cost control

Log every classification result — scores, reasoning, model version, and the action taken — even for allowed content. You'll need this history for appeals, policy tuning, and demonstrating compliance if you're regulated. For cost, batch non-urgent content (forum posts, async uploads) and reserve synchronous low-latency calls for things users wait on, like live chat. SubToAPI's dashboard shows token and request usage per key, which makes it straightforward to see which moderation category is driving most of your spend and adjust the pre-filter accordingly.

Scaling the pipeline

Once moderation is running in production, the operational concerns shift from "does classification work" to "can this handle load reliably." That means streaming for long documents (see /docs/streaming), rotating API keys per service or team without sharing secrets, and having usage metadata to catch sudden spikes that might indicate an attack rather than organic traffic growth. If you're running this on raw Claude access, you end up rebuilding key management and usage tracking yourself; SubToAPI wraps that into application API keys and a dashboard out of the box. You can get a working pipeline authenticated in minutes by following /docs/quickstart, and start on a free trial from /signup. Pricing for teams running moderation at scale is on /pricing.

questions

Does Claude block harmful content automatically, or do I need my own moderation layer? Claude has its own safety behavior, but that's not the same as enforcing your platform's specific policy. You still need a classification pipeline to apply your own taxonomy, thresholds, and escalation rules.

Should I classify every message or pre-filter first? Pre-filter with cheap deterministic checks first. Reserve Claude classification for content that's ambiguous or passes the first filter — this cuts API calls significantly without reducing accuracy.

How do I reduce false positives from over-aggressive thresholds? Log reasoning alongside scores, review blocked content weekly, and adjust per-category thresholds instead of a single global cutoff. Categories with high real-world cost for false negatives (self-harm) should stay strict; others (mild spam) can tolerate looser thresholds.