AI API production reliability: timeouts, retries, limits, and SLOs

Beginner: define success and unknown first

A synchronous JSON call succeeds after a valid final response. SSE succeeds only after its completion event. An asynchronous video succeeds at terminal completed with a valid result. HTTP 200, the first token, or a task ID are intermediate evidence. If a client times out without knowing whether upstream executed, mark the operation unknown and reconcile Request-ID, task ID, and usage logs before resubmitting.

Stage-based timeout budget

Split the business deadline across DNS and connection, TLS, request upload, first byte or token, stream idle, final response, task polling, and result download. The downstream deadline must fit inside the caller's remaining time and leave room for cleanup, settlement, and fallback.

Close bodies and propagate cancellation. File and media paths still need streaming I/O and byte limits; reliability code must not solve timeouts by buffering an entire request or response.

Retry decisions, backoff, and idempotency

Do not retry invalid requests, credentials, authorization, missing resources, balance, deterministic policy blocks, or business validation. A transient 429, connection reset, or selected 5xx is only a candidate when the operation is replay-safe and both per-request and global retry budgets remain. Honor Retry-After, otherwise use exponential backoff with full jitter.

Copyimport random, time

def retry_delay(attempt, retry_after=None, cap=20.0):
    if retry_after is not None:
        return min(float(retry_after), cap)
    return random.uniform(0, min(cap, 0.5 * (2 ** attempt)))

for attempt in range(4):
    result = call_once()  # bounded by attempt timeout and total deadline
    if result.ok or not result.retryable:
        break
    time.sleep(retry_delay(attempt, result.retry_after))

Read-only text is easier to replay than payment, messaging, or media creation. Write and async create operations need an application idempotency key and persisted state, plus a lookup before retry. Tool side effects are idempotent at the tool layer, not because the model promises not to repeat a call.

Rate, concurrency, queues, and backpressure

RPM and TPM bound request and token rates; concurrency protects connections, memory, and upstream in-flight capacity. Enforce tenant quota and model concurrency at admission. Every queue needs a maximum length, wait time, and rejection policy. Apply backpressure or cancellation to slow stream consumers and jitter async polling. Never create unbounded goroutines, promises, or in-memory queues during a spike.

Observe in-flight requests, queue depth, oldest wait, token-bucket capacity, 429 origin, and retry amplification. Retries consume capacity: 100 primary QPS with 0.3 average retries can deliver 130 QPS upstream.

Circuits, isolation, and tested fallback

Track health by channel, model, endpoint, and failure kind so one video provider does not break text. A circuit uses closed, open, and half-open states with minimum samples, failure rate, consecutive errors, and slow-call signals; half-open admits only probes. Authentication, balance, request errors, and content rejection do not count as supplier failures.

Prefer a tested sequence: reduce optional output or tools, move to a pinned family version, select a verified channel, select a verified fallback model, then return a recoverable error. Before switching models, verify fields, structured output, tools, safety, context, media format, price, and quality. Log requested model, effective model, and fallback reason.

SLOs, alerts, and cost attribution

Define availability by business terminal state. Measure TTFT, first audio, p50/p95/p99 completion, in-stream failure, task queue and generation time, cancellation, and unknown outcomes. Interactive, batch, and media workloads need separate SLOs. Alert on error-budget burn over time and minimum traffic, not one noisy percentage.

Correlate Request-ID, task ID, model, endpoint, channel, attempt, retry reason, state, usage, final charge, and fallback. Do not log keys, sensitive raw prompts, Base64, or signed URLs. Separate first attempts, retries, hedges, failed settlement, and refunds so higher apparent availability cannot hide doubled spend.

Expert: capacity tests and failure drills

Test steady traffic, bursts, long context, slow streams, client disconnect, upstream 429/5xx, DNS failure, proxy buffering, and permanently stuck tasks. Observe CPU, memory, pools, file descriptors, queues, Redis or database, bandwidth, and upstream concurrency. Use synthetic data and hard spend limits.

Drill circuit recovery, fallback failure, duplicate callbacks, object-storage failure, and delayed settlement. Acceptance criteria must quantify target throughput, p95/p99, maximum memory, error budget, unknown outcomes, retry amplification, recovery time, and billing variance. Re-run contract and capacity baselines after any model, channel, proxy, or timeout-policy change.