Build an AI Content Moderation Tool with Claude

Content moderation is a classification problem with high stakes: false negatives let harmful content through, false positives frustrate legitimate users. Claude is well suited to this because it can reason about context (sarcasm, cultural nuance, borderline cases) that keyword filters and older ML classifiers miss. This guide walks through building a working moderation tool with Claude, from designing the classification schema to handling edge cases and scaling it in production.

The core approach is straightforward: send user-generated content to Claude with a system prompt that defines your moderation categories, ask for a structured JSON response with a decision and confidence, and use that output to allow, flag, or block content. The hard part isn't the API call — it's designing categories that match your actual policy and handling the responses reliably at scale.

Step 1: Define your moderation categories

Before writing any code, decide what you're actually moderating. Generic "toxic/not toxic" labels are rarely useful in production. Most teams need something closer to:

• Harassment/hate — targeted abuse, slurs, threats
• Sexual content — explicit material, varying by platform policy
• Violence — graphic descriptions, threats of violence
• Spam/scam — phishing links, fake promotions
• Self-harm — content indicating risk to the user or others
• Off-topic/low-quality — not harmful, but not appropriate for the platform

Each category should map to an action: auto-block, auto-approve, or send to human review. This mapping is where most of your policy decisions actually live, not in the prompt.

Step 2: Design the prompt for structured output

Claude performs best on classification tasks when you ask for a fixed JSON schema and give it clear category definitions with examples of edge cases. A minimal system prompt looks like this:

You are a content moderation classifier. Given a piece of user-generated text,
classify it against these categories: harassment, sexual_content, violence,
spam, self_harm, none.

Return only valid JSON in this format:
{
  "flagged": boolean,
  "category": "harassment" | "sexual_content" | "violence" | "spam" | "self_harm" | "none",
  "confidence": number between 0 and 1,
  "reason": "short explanation, max 20 words"
}

Rules:
- If content fits multiple categories, choose the most severe.
- Sarcasm or fiction referencing violence should not be flagged unless it targets a real person.
- When uncertain, set confidence below 0.6 so it can be routed to human review.

Keep the category list short and mutually exclusive where possible. Long, ambiguous category lists lead to inconsistent classifications, which is worse than a slightly coarser schema applied consistently.

Step 3: Call the API and parse the result

Here's a working example using SubToAPI's messages endpoint, which gives you a standard HTTPS API on top of your Claude access with usage metadata included in every response:

async function moderateContent(text) {
  const response = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-5",
      max_tokens: 200,
      system: MODERATION_SYSTEM_PROMPT,
      messages: [{ role: "user", content: text }]
    })
  });

  const data = await response.json();
  return JSON.parse(data.content[0].text);
}

Since the response is JSON inside a text block, wrap parsing in a try/catch and have a fallback path (route to human review) if parsing fails. Models occasionally add stray text around the JSON, especially with lower max_tokens values, so keep the budget generous enough for a full response.

Step 4: Route decisions based on confidence and category

A production moderation pipeline typically has three lanes:

1. Auto-approve — flagged: false and confidence above your threshold
2. Auto-block — flagged: true, high-severity category (self_harm, harassment), confidence above threshold
3. Human review queue — everything with confidence below threshold, or medium-severity categories

function routeDecision(result) {
  if (!result.flagged) return "approve";
  if (result.confidence < 0.6) return "review";
  if (["self_harm", "harassment"].includes(result.category)) return "block";
  return "review";
}

Log every decision along with the raw Claude output. This gives you an audit trail and, over time, a labeled dataset you can use to tune thresholds or spot categories where the model is consistently wrong.

Step 5: Handle scale and cost

Moderation is often run on every message, comment, or upload, which means volume adds up fast. A few practical tips:

• Batch obviously safe content out early. Simple heuristics (length, known-safe patterns, allowlisted senders) can skip the API call entirely for a large fraction of traffic.
• Use streaming only where you need partial results. For moderation, a single blocking call is usually simpler and cheaper than streaming — see /docs/streaming if you do need incremental output for a related feature like live chat filtering.
• Track usage per surface. If you're moderating both comments and DMs, tag requests so you can see which surface drives cost and adjust category strictness accordingly.
• Set up separate API keys per environment. Running staging traffic through the same key as production makes usage and error debugging harder than it needs to be.

If you're building this as an internal tool rather than a customer-facing product, SubToAPI's dashboard gives you per-key usage tracking and team seats out of the box, so you don't have to build your own metering layer just to answer "how much did moderation cost us last month." Check /pricing for plan details, or /docs/quickstart to get an API key running in a few minutes.

Step 6: Test against real edge cases

Generic test suites won't catch the failure modes specific to your platform. Build a test set from your own historical data: known false positives, known false negatives, and borderline cases your moderators have flagged as ambiguous. Run this set against your prompt every time you change the category definitions, and track precision/recall per category rather than an aggregate score — a schema that's great at catching spam but weak on harassment will look fine in aggregate and fail where it matters.

questions

Can Claude replace a dedicated content moderation API entirely? For most mid-size platforms, yes for text — it handles nuance and context better than keyword filters. For image/video moderation or extremely high-volume, low-latency use cases (millions of requests per minute), you may want a hybrid approach with a fast pre-filter in front of Claude.

How do I keep moderation costs predictable? Use cheap heuristics to filter out obviously safe content before calling the API, cache repeated content hashes, and monitor usage per key so you catch volume spikes early. See /docs for usage metadata details.

What's the best way to reduce false positives? Give the model explicit examples of content that looks risky but shouldn't be flagged (sarcasm, fiction, quoted speech) directly in the system prompt, and route low-confidence results to human review instead of auto-blocking.