Claude API Content Moderation Use Case Guide

Content moderation is one of the most practical use cases for the Claude API: instead of maintaining brittle keyword filters or training a custom classifier, you send user-generated text (or images) to Claude with a moderation prompt and get back a structured verdict — flagged categories, severity, and a recommended action. This works because Claude is already good at understanding context, sarcasm, and intent, which is exactly where rule-based filters fail.

This article walks through how to design a moderation pipeline with the Claude API: what to send, how to structure the prompt and output, how to pick thresholds, and how to keep it fast and cheap enough to run on every piece of user content.

Why use an LLM for moderation instead of a classifier

Traditional moderation stacks combine a blocklist, a small ML classifier (like Perspective API), and human review queues. That works for high-volume, low-context content (spam, obvious slurs) but struggles with:

• Context-dependent harm — "I want to kill him" in a gaming chat vs. a threat
• Evolving evasion tactics — leetspeak, spacing, homoglyphs
• Multi-category nuance — content that's borderline harassment but fine as satire
• Multilingual content — most classifiers are English-first

Claude reads content the way a human reviewer would, applies your policy as written instructions, and can explain why it flagged something — which is invaluable for appeals and audit logs.

Designing the moderation prompt

The key is to give Claude a fixed taxonomy and force structured output so you can act on it programmatically. Don't ask "is this okay?" — ask for a JSON verdict against explicit categories.

You are a content moderation classifier. Analyze the following user-submitted
text against these categories: harassment, hate_speech, violence, sexual_content,
self_harm, spam. For each category, return a score from 0-3:
0 = not present, 1 = mild/ambiguous, 2 = clear violation, 3 = severe violation.

Return ONLY valid JSON in this shape:
{
  "categories": { "harassment": 0, "hate_speech": 0, "violence": 0,
                   "sexual_content": 0, "self_harm": 0, "spam": 0 },
  "max_score": 0,
  "action": "allow" | "flag_for_review" | "block",
  "reason": "one sentence explanation"
}

Text to review:
"""
{{user_content}}
"""

The action field lets you map straight to your business logic: allow publishes immediately, flag_for_review queues for a human moderator, block rejects outright. Tune the mapping (e.g., max_score >= 2 → block) based on your risk tolerance, not Claude's opinion alone — you own the policy, Claude just applies it.

Example request with SubToAPI

If you're already using SubToAPI to expose your Claude access as an HTTPS API, a moderation call looks like a normal messages request with a low max_tokens and temperature: 0 for consistency:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "max_tokens": 200,
    "temperature": 0,
    "messages": [
      {"role": "user", "content": "MODERATION_PROMPT_HERE"}
    ]
  }'

Because SubToAPI issues application-scoped keys (sub_live_...), you can create a dedicated key just for your moderation service, track its usage separately from your main product traffic, and rotate it independently if it's ever compromised. See /docs/quickstart for setup and /docs/messages for the full request reference.

Handling structured output reliably

Moderation pipelines need JSON you can parse without babysitting. A few practical rules:

• Set temperature: 0 — you want consistent, repeatable classifications, not creative variance
• Keep max_tokens small (150–250 is plenty for the schema above)
• Validate the response with a JSON schema parser before trusting it; if parsing fails, treat the content as flag_for_review rather than silently allowing it
• Log the raw response alongside your decision for audit trails and appeals

async function moderate(text) {
  const res = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "content-type": "application/json"
    },
    body: JSON.stringify({
      model: "claude-sonnet-4",
      max_tokens: 200,
      temperature: 0,
      messages: [{ role: "user", content: buildModerationPrompt(text) }]
    })
  });
  const data = await res.json();
  try {
    return JSON.parse(data.content[0].text);
  } catch {
    return { action: "flag_for_review", reason: "parse_error" };
  }
}

Batching, latency, and cost

Moderating every comment, DM, or upload in real time means volume adds up fast. A few patterns that help:

• Short-circuit with cheap filters first. Run a regex/blocklist pass before calling the API — obvious spam and profanity don't need an LLM call.
• Batch async content. For non-blocking flows (forum posts that don't need instant approval), queue content and moderate in batches rather than per-request.
• Cache repeat content. Hash and cache verdicts for identical or near-identical submissions (common with spam bots reposting the same text).
• Use streaming sparingly. Moderation doesn't benefit from streaming since you need the full JSON before acting — see /docs/streaming only if you're building a different, generative feature alongside moderation.

Image and multimodal moderation

Claude's vision capability extends the same pattern to uploaded images: send the image plus a moderation prompt asking about nudity, violence, or graphic content, and get the same structured verdict back. This is useful for platforms with image uploads (marketplaces, dating apps, community boards) where a text-only filter is useless.

Combining Claude with a human review layer

Even well-tuned prompts will produce edge cases. The most reliable production setups use Claude as a first-pass filter that handles the 90%+ of clearly obvious content (allow or block automatically) and routes ambiguous cases to human moderators. This keeps your team's time focused on the genuinely hard judgment calls instead of skimming thousands of obvious posts.

If you're evaluating whether to build this in-house against the Claude API directly or through a managed layer, check /pricing — for moderation workloads with steady volume, a fixed per-seat plan (Solo €9, Team €19/seat, Scale €49/seat) is often more predictable than raw token billing, especially once you factor in retries and prompt iteration during testing.

questions

Does Claude API moderation replace human moderators entirely? No — it's best used as a first-pass filter that auto-handles clear cases and routes ambiguous ones to human review, reducing volume rather than eliminating oversight.

How do I keep moderation costs predictable at scale? Filter obvious spam/profanity with cheap regex before calling the API, cache verdicts for repeat content, and consider a fixed-price plan like SubToAPI's Team or Scale tiers instead of raw per-token billing.

Can Claude moderate images as well as text? Yes, Claude's vision capability can review uploaded images against the same category-based prompt structure used for text, returning a comparable JSON verdict.