Audio, transcription, and Realtime API

Beginner: choose by transport

Text-to-speech uses POST /v1/audio/speech and normally returns MP3, WAV, or other audio bytes. File transcription and translation use multipart requests to POST /v1/audio/transcriptions and /v1/audio/translations. Bidirectional realtime audio uses the GET /v1/realtime WebSocket. These are binary HTTP, form HTTP, and a full-duplex connection; they cannot share one JSON or SSE parser.

Minimal TTS and transcription requests

Copycurl "$BASE_URL/v1/audio/speech" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini-tts","voice":"alloy","input":"Welcome to the audio API","response_format":"mp3"}' \
  --output speech.mp3

curl "$BASE_URL/v1/audio/transcriptions" \
  -H "Authorization: Bearer $API_KEY" \
  -F "model=gpt-transcribe" \
  -F "file=@meeting.wav" \
  -F "response_format=json"

Check status and Content-Type before writing speech output so a JSON error is not stored as an MP3. Validate upload format, byte length, duration, and channels. When splitting long recordings, preserve time offsets and reconcile overlapping segments.

Realtime WebSocket lifecycle

Connect to wss://<host>/v1/realtime?model=... with a site token. Configure a session, append audio, request a response, and consume server events according to the selected realtime model contract. Event names can differ across model protocols, so test the real model and channel. Never embed a long-lived key in a browser; use a constrained short-lived session or controlled backend proxy.

Copyconnect → session configured → audio append → response create
        ← transcript/audio deltas ← response done or error

Bound session duration, idle time, message size, outbound queue, and heartbeat. If a consumer falls behind, drop reconstructible visualization frames or terminate the session rather than allowing an unbounded memory queue.

Errors, disconnects, and duplicates

A 400 on file endpoints usually indicates format, duration, or field problems; 413 means the upload is too large. Realtime 401/403 occurs at the handshake, while later failures arrive as events. Record session ID, response ID, event sequence, and Request-ID. After disconnect, create a new session by default. Replay only from a server-confirmed checkpoint when the protocol explicitly supports recovery, or you can duplicate transcription, speech, or tool effects.

Track the playback cursor once audio reaches the user. On interruption, stop local playback, clear pending audio, and send a supported cancellation event. Obtain lawful consent for audio capture and explain what is recorded and processed.

Production quality, privacy, and cost

Evaluate accents, noise, overlapping speakers, domain terms, numbers, language switching, time to first audio, and interruption. Do not log raw audio by default. Keep minimal duration, model, state, and trace metadata; encrypt retained recordings, separate access, and enforce deletion. TTS, transcription, and Realtime can use different billing units, so reconcile the model marketplace and usage logs instead of assuming a single per-minute rule.

Expert: latency budget and capacity protection

Split latency into capture, network, VAD, upstream first byte, synthesis, jitter buffer, and playback SLOs. Use bounded buffers, streaming I/O, and per-connection limits. A slow consumer must receive backpressure or disconnection. Load-test concurrent WebSockets, audio frames per second, codec CPU, egress bandwidth, and reconnect storms, and ensure proxies neither buffer WebSocket traffic nor let one connection exhaust an instance.