How to Build a Content Moderation Tool with Claude

Building a content moderation tool with Claude means sending user-generated text (or images) to the model with a classification prompt, parsing a structured verdict back, and wiring that verdict into your app's moderation pipeline — block, flag for review, or allow. Unlike keyword-based filters, Claude understands context, sarcasm, and intent, which makes it good at catching the nuanced cases that regex and blocklists miss.

This guide walks through the actual implementation: how to design the prompt, how to get consistent structured output you can parse reliably, how to handle edge cases like borderline content, and how to put it behind an API so your app, your mobile client, and your internal review dashboard can all call the same moderation endpoint.

Why LLM-based moderation beats keyword filters

Traditional moderation stacks rely on blocklists, regex, and third-party classifiers trained on narrow categories. They fail in predictable ways:

• They can't tell "I want to kill this bug in my code" from an actual threat.
• They miss coded language, misspellings-as-evasion ("h4te speech"), and context-dependent harassment.
• They can't explain why something was flagged, which makes human review slower.

Claude reads the full context of a message — including surrounding conversation, tone, and intent — and returns not just a yes/no but a reasoned classification. That reasoning is what you'll show human moderators when something needs a second look.

Step 1: Define your moderation categories

Before writing any prompt, decide what you're actually filtering for. Generic "is this bad" prompts produce inconsistent results. Specific categories don't:

• Harassment / targeted abuse
• Hate speech / discrimination
• Violence or threats
• Sexual content (and whether it's allowed at all, or just restricted by context)
• Spam / scams
• Self-harm content
• PII leakage (emails, phone numbers, SSNs posted publicly)

Each category should have its own severity scale, because "block immediately" and "flag for review" need different downstream handling.

Step 2: Design the prompt for structured output

The key to a reliable moderation tool is forcing Claude to return parseable JSON every time, with no preamble. Here's a prompt structure that works well in production:

You are a content moderation classifier. Analyze the message below and
return ONLY valid JSON, no other text.

Categories: harassment, hate_speech, violence, sexual_content, spam, self_harm, pii

For each category, return a score from 0-3:
0 = not present, 1 = mild/ambiguous, 2 = clear violation, 3 = severe

Also return:
- "action": one of "allow", "flag_for_review", "block"
- "reason": one sentence explaining the highest-scoring category

Message to classify:
"""
{{user_message}}
"""

Return JSON only.

The 0-3 scale matters more than a binary flag — it lets you set different thresholds per category instead of a single global sensitivity dial. Harassment at level 1 might be fine in a gaming context; the same score for self-harm content should never be.

Step 3: Call the API and parse the response

Here's a working example using SubToAPI, which exposes Claude through a standard HTTPS endpoint with an API key — useful if you want moderation running as its own service rather than embedded in your main app code.

async function moderateContent(message) {
  const response = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4",
      max_tokens: 300,
      messages: [
        {
          role: "user",
          content: `You are a content moderation classifier... [prompt above] Message: """${message}"""`,
        },
      ],
    }),
  });

  const data = await response.json();
  const text = data.content[0].text;
  return JSON.parse(text);
}

const result = await moderateContent("some user comment here");
// { harassment: 0, hate_speech: 0, violence: 0, sexual_content: 0,
//   spam: 0, self_harm: 0, pii: 0, action: "allow", reason: "..." }

Wrap the JSON.parse in a try/catch — occasionally a model returns a stray character outside the JSON block, and you don't want that to crash your moderation pipeline. On parse failure, default to flag_for_review rather than allow, so bad output fails safe.

Step 4: Set thresholds and route actions

Once you have scores, map them to actions in your own code rather than trusting the model's action field blindly for anything safety-critical:

function decideAction(scores) {
  if (scores.self_harm >= 2 || scores.violence >= 3) return "block";
  if (scores.harassment >= 2 || scores.hate_speech >= 2) return "flag_for_review";
  if (scores.pii >= 1) return "flag_for_review";
  return "allow";
}

This separation matters because thresholds will change as you tune the system — you don't want to re-prompt every time you adjust a cutoff.

Step 5: Handle streaming and high volume

For moderation gates that run before a message is posted (not after), latency matters. Claude's non-streaming response is usually fast enough for single-message checks, but if you're moderating long documents or batches, consider streaming the response and bailing out early if the first tokens indicate a severe violation — see /docs/streaming for how response streaming works. For bulk backfills (moderating historical content), batch requests sequentially or in small concurrent groups rather than firing thousands of parallel calls at once.

If your moderation tool needs to call out to external systems — a user reputation lookup, a ban-list check — tool use lets Claude request that data mid-classification instead of you pre-fetching everything. Details are in /docs/tools.

Step 6: Log everything for audits

Moderation decisions get disputed. Store the full request, the raw model response, the parsed scores, and the final action taken — not just a boolean. When a user appeals a ban, you want to show exactly what triggered it and reproduce the decision if needed. SubToAPI's dashboard includes usage metadata per request, which helps when you need to trace a specific classification back to its API call without building separate logging infrastructure.

Getting started

To build this yourself: sign up at /signup for a free trial, get a sub_live_ key, and follow /docs/quickstart to send your first classification request. The /docs/messages reference covers the full request/response shape if you want to extend the prompt with more categories or multi-turn context (e.g., classifying a reply in light of the thread it's responding to). Pricing for ongoing use is at /pricing — Solo at €9 covers most single-app moderation workloads; Team and Scale add seats for larger moderation and trust & safety teams.

FAQ

Is Claude accurate enough to replace a dedicated moderation API? For nuanced, context-dependent cases (sarcasm, coded language, intent), yes, it generally outperforms keyword and traditional classifier approaches. For narrow, high-volume categories like spam detection, a cheaper specialized classifier as a first pass combined with Claude for ambiguous cases is often more cost-effective.

How do I prevent the model from being jailbroken into approving bad content? Keep the moderation prompt separate from any user-controllable instructions, never let the user message include text that looks like system instructions, and always treat the content being classified as untrusted data inside clearly delimited triple-quotes, as shown above.

Should moderation run before or after content is posted? Run it before (synchronously, blocking the post) for anything legally or safety-critical, like self-harm content. For lower-stakes categories like mild spam, post-then-review is often enough and keeps your app feeling responsive.