How to Build a Claude-Powered Content Moderation Tool

Building a content moderation tool with Claude means sending user-generated text (or transcribed audio, OCR'd images, etc.) to the model with a structured prompt that returns a classification — flagged or not, which policy category, and a confidence or severity score. Unlike keyword-based filters, Claude understands context, sarcasm, and intent, which makes it far better at catching the edge cases that regex and blocklists miss, while still being controllable enough to run in a production pipeline.

This guide walks through the actual architecture: how to design the moderation prompt, how to structure the output so your application can act on it programmatically, how to handle scale and cost, and how to expose it as a stable API your app or your team can call.

Why Claude for moderation instead of a classifier model

Dedicated moderation classifiers (perspective-style APIs) are fast and cheap but brittle — they score toxicity on a single axis and struggle with nuance like reclaimed slurs, satire, or coded harassment. Claude gives you:

• Policy-aware reasoning — you can describe your actual community guidelines in the prompt, not just "toxic/not toxic."
• Multi-category output — harassment, self-harm, spam, scams, hate speech, sexual content, violence, all scored independently.
• Explanations — useful for human review queues and appeals.
• Multilingual coverage — without training separate models per language.

The tradeoff is latency and cost per call compared to a lightweight classifier, so most production moderation tools use Claude as the second-stage, context-aware check after a cheap first-pass filter (blocklist, rate limiting, or a simple heuristic) catches the obvious stuff.

Designing the moderation prompt

The key is forcing structured, parseable output. Don't ask Claude to "moderate this" — give it explicit categories and require JSON back.

SYSTEM PROMPT:
You are a content moderation classifier. Given a piece of user text,
return ONLY valid JSON with this shape:

{
  "flagged": boolean,
  "categories": {
    "harassment": 0-1,
    "hate_speech": 0-1,
    "self_harm": 0-1,
    "sexual_content": 0-1,
    "violence": 0-1,
    "spam": 0-1
  },
  "action": "allow" | "review" | "block",
  "reason": "short explanation, max 20 words"
}

Score each category by severity, not just presence. Mild profanity
is not harassment. Satire and criticism of public figures is not
hate speech unless it targets a protected characteristic.

Keep the scoring scale consistent (0–1 floats) so you can set numeric thresholds in code rather than branching on free text. Decide your action thresholds upfront — e.g. anything above 0.8 in any category auto-blocks, 0.4–0.8 goes to a human review queue, below 0.4 is allowed.

Calling the model

Here's a minimal request/response loop using the Messages API through SubToAPI, which exposes Claude as a standard HTTPS endpoint with an application API key — useful if you want moderation running behind your own backend without managing Anthropic auth or SDK versions directly:

async function moderateText(text) {
  const res = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4",
      max_tokens: 300,
      system: MODERATION_SYSTEM_PROMPT,
      messages: [{ role: "user", content: text }],
    }),
  });

  const data = await res.json();
  return JSON.parse(data.content[0].text);
}

Full request/response schema is in the Messages docs. If you're new to the API, the quickstart covers auth and your first call in under five minutes.

Handling scale: batching and streaming

If you're moderating a firehose (chat messages, comments, live transcripts), two things matter:

1. Throughput — batch short texts into a single call where context allows, or parallelize calls with a queue and concurrency limit instead of serial requests.
2. Latency for live chat — for real-time moderation in chat apps, you often want the allow/block decision fast enough to not visibly delay the message. Keep max_tokens low (moderation JSON is small) and avoid unnecessary system prompt bloat. If you're layering moderation into a broader streaming response pipeline, the streaming docs explain how to handle partial output correctly.

For tool-heavy moderation flows — e.g. the model decides whether to call a "flag_for_review" function versus "allow" — structured tool calling is cleaner than parsing freeform JSON. See the tools docs for defining a moderation decision as a callable tool with a strict schema, which eliminates most JSON-parsing failures.

Building the review pipeline

A moderation tool is incomplete without a human-in-the-loop layer for ambiguous cases:

• Log every decision — text, categories, action, model version, timestamp. You'll need this for audits and for tuning thresholds later.
• Queue "review" actions — don't auto-block on medium confidence; surface to a moderator dashboard.
• Track false positive/negative rates — sample reviewed content periodically and compare human verdict to Claude's output to catch threshold drift.
• Version your prompts — treat the system prompt like code; changing wording changes behavior, so store prompt versions alongside logged decisions.

Cost and reliability in production

Moderation calls are typically short (both input and output), so per-call cost is low, but volume adds up fast on high-traffic platforms. Practical cost controls:

• Pre-filter with cheap heuristics before calling Claude at all.
• Cache decisions for identical or near-identical repeated content (spam often repeats).
• Use a smaller/faster model tier for first-pass scoring and escalate only borderline cases to a stronger model.

On the infrastructure side, running moderation through a stable API key setup — rather than embedding provider credentials across multiple services — makes key rotation, usage tracking per team or per product, and rate-limit visibility much easier to manage. SubToAPI's pricing starts at Solo (€9) for individual projects, with Team (€19/seat) and Scale (€49/seat) tiers adding seats and usage dashboards for teams running moderation across multiple products. You can test the full flow with a free trial at signup.

questions

Can Claude replace a dedicated moderation API entirely? It can for most text use cases, especially where policy nuance matters, but pairing it with a cheap first-pass filter reduces cost and latency for high-volume traffic.

How do I stop Claude from returning invalid JSON? Use a strict system prompt with an exact schema, set low max_tokens, and consider using tool calling (see the tools docs) instead of freeform text parsing — it enforces structure at the API level.

What threshold should I use for auto-blocking vs. human review? There's no universal number — start around 0.8+ for auto-block and 0.4–0.8 for review, then tune based on logged false positive/negative rates from real traffic.