Model evaluation, canarying, and version migration

Beginner: define good first

Sample successful, edge, and failed production tasks and remove sensitive data. Give every case a judgeable expectation: a factual answer, JSON schema, citation, tool arguments, image subject, transcript, or refusal policy. Without an expected outcome, an upgrade cannot be distinguished from sampling noise.

Copy{"case_id":"support-042","input_version":"v3","model":"<PINNED_ID>","expected":{"schema_valid":true,"must_include":["refund terms"]},"metrics":["quality","latency_ms","cost"]}

Establish the current baseline

Record exact model ID, system prompt, parameters, tool schema, retrieval version, and input digest. Repeat stochastic tasks enough to estimate success, format pass rate, p50 and p95 latency, error rate, and cost distribution. For image, audio, and video, use blind human review or task metrics rather than choosing one attractive sample.

Compare candidates

Run the same offline set first and isolate safety, tool, long-context, and multilingual regressions. After thresholds pass, use shadow traffic or a small canary. Experimental calls must not execute real payments or external writes. The same model name through another channel still requires evaluation because mapping, region, and version may differ.

Lifecycle and migration

Distinguish stable, preview, open weights, and legacy. Preview needs an exit path. Legacy needs the official notice, shutdown date, replacement, and owner. A moving alias and a fixed snapshot carry different risks. Validate endpoint, fields, stream events, tools, token accounting, and safety before changing more than the model string.

Canary, rollback, and observation

Route by tenant or stable hash and keep the old model ready. Observe quality proxies, complaints, p95 and p99 latency, 429 and 5xx, stream interruption, usage, and cost per task. Roll back at a defined stop threshold and keep a dual-version reconciliation period after rollout.

Expert continuous evaluation

Gate prompt, route, and price changes with reproducible evaluations and snapshots. Add frequent failure cases without overfitting to the suite. For agents, retain each tool decision and side effect. For RAG, separate recall, reranking, and generation metrics. Evidence must answer which change caused which regression.