AI gateway admin runbook: channel rollout, retries, and supplier health

Beginner: locate the four control surfaces

/console/channel manages channels, models, weights, and tests. /console/channel-health shows success, errors, and temporary state. /console/adapter-workbench manages suppliers, accounts, catalog scans, and pricing. /console/setting contains global retry and storage settings. These are protected administration surfaces, not public AI client APIs.

Channel routes require an operator identity plus channel permission; batch image-storage changes require admin access. Reading a single channel secret additionally requires root access, a critical-operation rate limit, no-cache handling, and secure verification. Never place secrets in tickets, screenshots, URLs, query strings, or ordinary logs.

Preflight configuration

Keep the new channel isolated. Verify that the channel type matches its adapter, base_url is a trusted HTTPS origin, the credential is least-privileged, models reflects real upstream availability, and model_mapping points from the public name to the upstream name. Check group, priority, and weight before separate tool, stream, image, audio, and async tests.

Copy{
  "name": "provider-canary",
  "type": "match-the-adapter",
  "base_url": "https://trusted-upstream.example/v1",
  "models": "model-a,model-b",
  "model_mapping": {"public-model-a": "model-a"},
  "group": "canary",
  "priority": 0,
  "weight": 0
}

This is a review shape, not a complete submission. Keep the secret out of documentation and automation logs, and keep the channel outside production routing until validation finishes.

Channel test, canary, and rollback

Use POST /api/channel/test/{id}/run for a selected model and protocol. The legacy GET /api/channel/test/{id} also exists, but the newer runner expresses a richer target. A passing sample does not replace tests for the real endpoint, long context, streaming, tools, and media.

Canary through a small fixed share or isolated group and record actor, time, model, price, and rollback criteria. Observe /api/channel/success_rate, /api/channel/error_stats, and 24-hour call counts. Restore the prior weight or status when error, p95 latency, billing variance, or quality crosses its gate. POST /api/channel/{id}/reset_temporary_state clears temporary routing state; it does not repair credentials, balance, or protocol.

Retry and replay boundaries

Normal relay policy starts from global RetryTimes and RetryTimeoutSeconds; model metadata can override them with max_retries and retry_timeout_seconds. The effective policy is frozen when the request begins. Zero retries disables retry. A zero retry-time budget means no additional cumulative retry limit, not an infinite request timeout.

By default, the gateway does not retry 400, 408, 504, 524, a response already sent downstream, a fixed channel request, an explicitly non-retryable error, or a canceled client request. Configurable status ranges only identify candidates. Media creation, tool side effects, and any operation the upstream may have accepted still require replay-safety decisions. A bounded Retry-After and the remaining client deadline also gate the next attempt.

Cooldown, automatic disablement, and recovery

In-process health protection defaults to three consecutive failures and a 30-second cooldown. CHANNEL_HEALTH_FAILURE_THRESHOLD and CHANNEL_HEALTH_COOLDOWN_SECONDS adjust these values; success clears the state. This is process-local state, not a durable cluster-wide circuit breaker. Routing policy and channel priority can also change recovery-probe timing.

Automatic disablement is separate and its default status set includes 401. Repair credentials for authentication errors, reconcile supplier accounts for balance errors, and repair mapping for request errors. Do not hide the cause by resetting state or allowing unlimited retries. Recover with one narrow probe before gradual traffic.

Supplier, account, and catalog health

A supplier catalog says what upstream advertises; a channel test says whether the current credential and protocol work; the marketplace says what this site exposes. Use /api/suppliers/catalog/supplier-health for catalog-scan health and /api/supplier_accounts/health/summary, latest balance, trend, and account events for credential or funding issues.

Catalog scans, price sync, and mapping are management changes. Preview differences, confirm origin, snapshot, unit, and currency, then apply. Alert destinations, secret reveal, credential writes, and batch cutover have higher privilege and need actor, approval, and result records.

Expert: observability, fault tree, and handoff

Correlate Request-ID, requested and mapped models, channel ID, supplier account, attempt, status, error class, upstream request ID, usage, price version, and final settlement. Classify first: all-channel failure suggests ingress, model, or policy; one channel suggests credential, region, or protocol; one account suggests balance or suspension; one model suggests mapping or lifecycle; stream/media-only failure suggests its specialized protocol or result storage.

The handoff record should name the owner, data region, quota, balance threshold, supported endpoints, allowed models, rollback action, and upstream status page. Re-run contract tests after a credential, base URL, mapping, retry, or weight change. Keep raw secrets, prompts, Base64, signed URLs, and confidential supplier costs out of ordinary telemetry labels.