How to Build a Content Moderation Tool with Claude

Building a content moderation tool with Claude means sending user-generated text (or image descriptions) to the model with a structured prompt that asks it to classify the content against defined policy categories, then parsing the response into a decision your application can act on — allow, flag, or block. Unlike keyword-filter or regex-based moderation, Claude understands context, sarcasm, and intent, which makes it far better at catching nuanced policy violations while reducing false positives on legitimate content.

This guide walks through the actual implementation: how to design the classification prompt, how to get consistent structured output, how to handle edge cases like borderline content, and how to wire it into a real pipeline with rate limits and audit logs.

Why Claude for Moderation Instead of a Classic Filter

Rule-based filters (banned word lists, regex patterns) fail on two fronts: they miss obfuscated or contextual violations ("how do I hurt someone" rephrased creatively), and they over-flag innocent content (a nurse discussing wound care gets blocked for "blood"). An LLM-based moderator reads the whole message, understands what the user actually means, and can apply nuanced judgment against your specific policy — including severity levels, not just binary allow/block.

The tradeoff is latency and cost per request, which is why most production systems combine both: a fast regex pre-filter for obvious cases, and Claude for everything that needs real judgment.

Step 1: Define Your Moderation Categories

Before writing any code, write down the categories you actually care about. Generic "is this bad" prompts produce inconsistent results. Be specific:

• Harassment or targeted abuse
• Hate speech or discrimination
• Sexual content involving minors (always hard-block, no review needed)
• Violence or graphic threats
• Self-harm content
• Spam or scams
• PII leakage (phone numbers, addresses, card numbers)

Each category should map to an action: block, flag_for_review, or allow. This mapping is a business decision, not a model decision — Claude tells you what it sees, your code decides what happens next.

Step 2: Design the Classification Prompt

Ask Claude to return structured JSON so your application can parse it reliably without fragile string matching.

const SYSTEM_PROMPT = `You are a content moderation classifier. Analyze the user message
against these categories: harassment, hate_speech, sexual_minors, violence,
self_harm, spam, pii_leak.

For each category, return whether it applies and a severity score 0-3
(0 = none, 1 = mild, 2 = moderate, 3 = severe).

Respond ONLY with valid JSON in this exact shape:
{
  "categories": {
    "harassment": {"flagged": boolean, "severity": number},
    "hate_speech": {"flagged": boolean, "severity": number},
    "sexual_minors": {"flagged": boolean, "severity": number},
    "violence": {"flagged": boolean, "severity": number},
    "self_harm": {"flagged": boolean, "severity": number},
    "spam": {"flagged": boolean, "severity": number},
    "pii_leak": {"flagged": boolean, "severity": number}
  },
  "overall_action": "allow" | "flag_for_review" | "block",
  "reason": "short explanation"
}`;

Keep the system prompt stable across requests — this is what makes scoring consistent over time. Don't let users or support agents edit it per-request.

Step 3: Call the API and Parse the Result

Using SubToAPI's Messages endpoint, the request looks like any other Claude call, just with the moderation system prompt attached:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet",
    "max_tokens": 300,
    "system": "You are a content moderation classifier...",
    "messages": [
      {"role": "user", "content": "The user message to moderate goes here"}
    ]
  }'

Parse the response, validate it against your schema, and apply your action mapping:

const result = JSON.parse(response.content[0].text);

if (result.overall_action === "block") {
  rejectSubmission(result.reason);
} else if (result.overall_action === "flag_for_review") {
  queueForHumanReview(result);
} else {
  publishContent();
}

Always wrap the JSON parse in a try/catch. Models occasionally wrap output in explanatory text even when told not to — set max_tokens conservatively and consider a retry with a stricter reminder if parsing fails.

Step 4: Handle Severity Thresholds, Not Just Flags

A binary flagged/not-flagged system either over-blocks or under-blocks. Use severity scores to build tiers:

• Severity 0-1: allow automatically
• Severity 2: flag for human review, allow provisionally or hold depending on category
• Severity 3: block immediately, no review needed for hard categories like sexual content involving minors

This gives your moderation team a queue sorted by real risk instead of a flat pile of flagged items.

Step 5: Add a Fast Pre-Filter

Calling an LLM on every single message is slow and costs money at scale. Add a cheap pre-filter in front of Claude:

• Obvious profanity list → auto-flag without calling the API
• Message length under 3 characters → skip moderation, allow
• Known spam domains in URLs → auto-block

Route everything else to Claude. This cuts your API call volume significantly while keeping nuanced judgment where it matters.

Step 6: Log Everything for Audit and Tuning

Store the raw input, the full classification JSON, and the final action taken for every moderation decision. This is non-negotiable for two reasons: compliance (you need to show why content was removed if challenged) and tuning (you'll periodically review false positives/negatives and adjust your prompt or thresholds).

If you're running this across a team — moderators, engineers reviewing edge cases, and admins managing policy — a dashboard with usage metadata per API key helps you see which part of your pipeline is generating the most flagged content and whether your pre-filter is actually reducing load. SubToAPI's dashboard gives you request logs and per-key usage out of the box, so you're not building that observability layer from scratch. See the quickstart to get an API key, or check pricing if you're scaling across a moderation team.

Step 7: Test Against Known Edge Cases

Build a test set before shipping: clearly bad content, clearly fine content, and the hard middle — sarcasm, medical/legal discussion that touches sensitive topics, quoted content ("he said X" vs. actually saying X). Run this set against your prompt every time you change it. Treat it like a regression suite, because small prompt edits can shift behavior across the whole category set.

Putting It Together

A production moderation pipeline is: cheap pre-filter → Claude classification with structured JSON → severity-based routing → human review queue for borderline cases → audit log. Claude handles the judgment call; your code handles the policy enforcement. Review the Messages API docs for request/response details, and if you need moderation running on near-real-time streams of content, the streaming guide covers token-by-token responses if your UI needs to show results as they arrive.

questions

Does Claude moderation replace human moderators entirely? No. Use it to triage — auto-allow clearly safe content, auto-block clearly severe violations, and route the ambiguous middle to human reviewers. This cuts human review volume by a large margin without removing oversight on hard cases.

How do I keep moderation costs predictable at scale? Add a regex/keyword pre-filter to skip obvious cases, cache repeated content hashes, and monitor usage per API key so you can see cost trends before they become a surprise. A dashboard with per-key usage metadata makes this easy to track.

Can Claude moderate images, not just text? Yes, Claude accepts image inputs and can classify visual content the same way — describe your categories in the system prompt and send the image alongside the user's text for context.