Claude API Content Moderation Example for Developers

Content moderation with Claude works by sending user-generated text (or images) to the model with a prompt that asks it to classify the content against your policy categories, then parsing the structured response in your application. Unlike a dedicated moderation endpoint, Claude gives you flexible, customizable judgment — you define the categories, the severity scale, and the action thresholds yourself.

This guide walks through a working example: a moderation function that takes arbitrary text input, returns a structured verdict (allowed, flagged, blocked), a confidence score per category, and a short explanation — all in one API call you can drop into a signup flow, comment section, or chat app.

Why use Claude instead of a keyword filter

Keyword and regex filters catch obvious terms but miss context. "I want to kill this bug in my code" and "I want to kill my neighbor" both contain "kill," but only one is a threat. Claude reads intent, sarcasm, coded language, and multi-sentence context, which is why most teams layer an LLM check on top of (not instead of) cheap pre-filters.

The tradeoff is latency and cost per call — you don't want to run a full LLM check on every keystroke. The common pattern is: cheap filter first, Claude moderation on anything ambiguous or on all submitted content before it goes live.

Designing the moderation prompt

The core trick is forcing structured, parseable output. Give Claude your exact category list and ask for JSON only — no prose, no hedging.

You are a content moderation system. Classify the following user-submitted
text against these categories: harassment, hate_speech, violence, sexual_content,
self_harm, spam. For each category return a score from 0 (not present) to 1
(clearly present). Also return an overall "action" field: "allow", "review",
or "block". Respond with JSON only, no other text.

Text: """{{user_input}}"""

Claude's instruction-following is strong enough that with this prompt, you consistently get clean JSON you can JSON.parse() without extra cleanup logic in most cases. Still, always wrap parsing in a try/catch and have a fallback (treat unparseable responses as "review").

A working moderation function

Here's a complete example using the Messages API through SubToAPI, which exposes Claude behind a standard HTTPS endpoint with application API keys — useful if you want usage metadata per moderation call without managing Anthropic console credentials directly.

async function moderateContent(text) {
  const response = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4",
      max_tokens: 300,
      messages: [
        {
          role: "user",
          content: `You are a content moderation system. Classify the
following text against these categories: harassment, hate_speech, violence,
sexual_content, self_harm, spam. Score each 0-1. Return an "action" field:
"allow", "review", or "block". Respond with JSON only.

Text: """${text}"""`,
        },
      ],
    }),
  });

  const data = await response.json();
  const raw = data.content[0].text;

  try {
    return JSON.parse(raw);
  } catch {
    return { action: "review", error: "unparseable_response", raw };
  }
}

A typical response looks like:

{
  "harassment": 0.1,
  "hate_speech": 0.0,
  "violence": 0.0,
  "sexual_content": 0.0,
  "self_harm": 0.0,
  "spam": 0.85,
  "action": "block"
}

Your application logic then branches on action: allow publishes immediately, review routes to a human queue, block rejects with a user-facing message.

Tuning thresholds per category

Don't treat every category the same. A hate-speech score of 0.3 might warrant an automatic block, while a spam score of 0.3 might just mean "watch this user." Build a small config object instead of hardcoding logic:

const thresholds = {
  hate_speech: 0.3,
  violence: 0.4,
  sexual_content: 0.5,
  self_harm: 0.2,
  harassment: 0.4,
  spam: 0.7,
};

function evaluate(scores) {
  for (const [category, limit] of Object.entries(thresholds)) {
    if (scores[category] >= limit) return "block";
  }
  return "allow";
}

This lets product and legal teams adjust sensitivity without touching prompts or redeploying model logic — just update a config file.

Moderating images and screenshots

Claude's vision capability extends this same pattern to uploaded images: profile pictures, screenshots, memes. Send the image as base64 or a URL in the same message structure, same JSON-output instruction, same category list. This catches things a text-only filter can't: a slur embedded in an image, a violent meme with no caption, or NSFW content uploaded with a blank description.

Batching and cost considerations

Running every comment through a full moderation call adds up. A few practical patterns that keep costs sane:

• Pre-filter cheap and obvious cases (profanity lists, URL spam patterns) before calling Claude.
• Batch borderline items and moderate them together in one request when latency isn't critical (e.g., async content review queues).
• Cache verdicts for identical or near-identical repeated submissions (common with spam bots posting the same text repeatedly).
• Track usage per endpoint so you know moderation calls aren't silently becoming your biggest API cost line. SubToAPI's dashboard shows per-key usage, which is useful for separating moderation traffic from your main product traffic in billing.

If you're prototyping this, start with the quickstart to get an API key, then check the messages docs for the full request/response schema you'll be parsing against.

Handling false positives gracefully

No moderation system is perfect. Always give users a path to appeal a block, log the full model response (not just the action) so you can audit decisions later, and periodically sample "allow" verdicts to check for missed violations. Treat the category scores as signals for a decision pipeline, not as ground truth — especially for borderline "review" cases where a human should have the final say.

Questions

Can Claude moderation replace a dedicated moderation API entirely? It can for most text and image use cases, especially when you need custom categories a generic moderation endpoint doesn't support. For extremely high-volume, low-latency needs (like per-keystroke filtering), pair it with a cheap pre-filter.

How do I keep moderation output consistent across requests? Use a strict JSON-only prompt, low temperature settings, and validate the response schema in code. Treat any malformed response as "review" rather than silently allowing it.

Does streaming work for moderation calls? You generally don't need streaming for moderation since you want the full structured verdict before acting — see streaming docs if you have a use case that benefits from partial results anyway.