Content Moderation safety classification API

Beginner: Moderation is a risk classifier, not the final decision

POST /v1/moderations accepts input and returns flagged, categories, and category_scores for each item. The gateway has a compatibility default when model is omitted, but production calls should explicitly name a verified omni-moderation-latest or available pinned snapshot so a default cannot change unnoticed. Moderation covers its published taxonomy; account authorization, child safety, privacy, copyright, regulated-industry, and community rules remain business policy.

Minimal request and response handling

Copycurl "$BASE_URL/v1/moderations" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "omni-moderation-latest",
    "input": ["First user text to inspect", "Second item to inspect"]
  }'

results[] follows input order. Validate length and required fields before reading flagged. Retain category booleans and scores for policy and evaluation, but never present a score as a certain probability. Batching can reduce overhead while expanding failure scope and response size, so bound items, characters, tokens, and request bytes.

Policy layers and human review

Map classifications to explicit actions such as allow, ask for revision, reduce distribution, block publication, quarantine, review, or emergency escalation. High-risk categories can block immediately while boundary scores enter review. One global threshold will not fit every language, region, and product. A review UI should expose only necessary context, restrict access, and record view, decision, appeal, and override reason.

Inspect both input and output. Tool arguments, RAG documents, image text, and user-generated media can bypass a policy that checks only the final answer. For multimodal input, verify that the selected model and gateway channel accept the exact content shape; upstream support alone is not a gateway promise.

Errors, deadlines, and degradation

A 400 usually means an invalid input shape, content type, or model. 401/403 concerns credentials or permission. 429 can indicate rate, balance, or quota, while a transient 5xx may receive finite backoff. On timeout, apply an approved fail-open, fail-closed, or human-review policy based on risk. Never silently allow everything during an outage or retry forever on the request path.

Log Request-ID, policy version, model, content hash, classification summary, action, review state, and latency without storing raw sensitive content by default. Include normalized content hash, model, and policy version in short-lived cache keys so policy updates do not reuse an old decision.

Expert: calibration, drift, and governance evidence

Evaluate per-category precision, recall, false blocks, misses, review volume, and appeal reversals on labeled data covering language, spelling variation, context ambiguity, and adversarial bypass. Shadow and canary changes behind a latest alias while monitoring distribution and score drift. Version model output, business policy, and human decision separately so a historical action is explainable. Rehearse provider outage, fallback, review backlog, and sensitive-data deletion so safety controls produce evidence rather than remaining principles on paper.