Streaming audio over WebSockets

The interview version of “experience with WebSockets and real-time communication (streaming data or audio over persistent connections)” is a design question: a browser or a phone line sends audio, the server transcribes it as it arrives, an LLM answers, and synthesised speech comes back — all on one connection, with the user able to interrupt.

This note is that pipeline in FastAPI, and what breaks in it. The protocol itself is in WebSockets.

The shape

text

client                        server
  │ {"type":"start",rate,enc}   │
  ├────────────────────────────▶│ validate, open STT
  │ binary PCM frames (20 ms)   │
  ├────────────────────────────▶│ buffer → VAD → STT
  │◀────────────────────────────┤ {"type":"partial",text}
  │◀────────────────────────────┤ {"type":"final",text}
  │◀────────────────────────────┤ TTS frames + "mark"
  │ {"type":"interrupt"}        │
  ├────────────────────────────▶│ cancel TTS task
  │ {"type":"end"}              │
  ├────────────────────────────▶│ flush, close 1000

Two channels on one socket: JSON text frames for control (start, end, interrupt, transcripts, marks) and binary frames for audio. Base64 audio inside JSON works — telephony providers do it — at 33% more bytes and a decode per frame; binary frames are the default when you own both ends.

The start message carries the sample rate, encoding and channels, because the bytes do not (Audio fundamentals for backend engineers).

The server

python

import asyncio, json
from fastapi import FastAPI, WebSocket, WebSocketDisconnect

app = FastAPI()
FRAME = 640                                   # 20 ms @ 16 kHz int16 mono

@app.websocket("/ws/transcribe")
async def transcribe(ws: WebSocket):
    await ws.accept()
    start = await ws.receive_json()
    if (start.get("rate"), start.get("encoding")) != (16000, "pcm_s16le"):
        await ws.close(code=1003, reason="expected 16 kHz pcm_s16le"); return

    frames: asyncio.Queue[bytes | None] = asyncio.Queue(maxsize=250)  # ~5 s

    async def reader():                       # socket -> queue
        buf = PcmBuffer(FRAME)
        try:
            while True:
                msg = await ws.receive()
                if msg.get("bytes"):
                    for frame in buf.push(msg["bytes"]):
                        await frames.put(frame.tobytes())   # backpressure here
                elif msg.get("text") and json.loads(msg["text"])["type"] == "end":
                    return
                elif msg["type"] == "websocket.disconnect":
                    return
        finally:
            await frames.put(None)            # sentinel: flush and finish

    async def transcriber():                  # queue -> STT -> socket
        async for ev in stt.stream(iter_queue(frames), rate=16000):
            await ws.send_json({"type": "final" if ev.is_final else "partial",
                                "text": ev.text})

    try:
        await asyncio.gather(reader(), transcriber())
    except WebSocketDisconnect:
        pass

The structure is the whole answer: one task reads the socket, one task drives the vendor stream, a bounded queue sits between them. The reader never does slow work, so a slow STT never stalls the socket’s ping/pong and drops the connection.

The queue bound is the backpressure — if the consumer falls five seconds behind, put blocks the reader, TCP fills, and the client’s sends slow down instead of the server’s memory growing. The sentinel tells the consumer to flush the recogniser’s last partial and emit the final.

Interim results are display-only; downstream logic keys on final (Speech: STT and TTS). The vendor stream is the provider’s streaming client — Deepgram, AssemblyAI, a self-hosted Whisper server — consuming an async iterator of frames.

Chunk size, latency and the budget

Frame	Trade
10–20 ms	lowest latency; ~50 messages/s of overhead
50–100 ms	fewer messages; adds up to 100 ms before the first byte
500 ms+	batch-like; the conversation feels laggy

20 ms is the telephony convention and what VADs expect; 40–100 ms is a fine wire size for a browser client that buffers locally. What matters is that the frame size is fixed and announced, and that nothing on the path waits for “enough” audio before forwarding.

The end-to-end budget for a conversation — capture, transport, STT finalisation, LLM first token, TTS first byte — is in Realtime voice agents; the WebSocket’s share is tens of milliseconds unless something buffers.

The reply path and barge-in

The response goes back the same way: TTS audio as binary frames, in the format the start message declared for playback, with a JSON mark after each sentence so the client can report how far it has played.

Barge-in is then a protocol: the client detects speech (or the server’s VAD does), sends interrupt, the server cancels the TTS task and the LLM stream, and the last mark the client acknowledged tells the conversation state what the user actually heard.

python

speaking: asyncio.Task | None = None

async def on_control(msg: dict):
    global speaking
    if msg["type"] == "interrupt" and speaking and not speaking.done():
        speaking.cancel()                     # stop sending audio now
        await ws.send_json({"type": "cleared"})   # client flushes its buffer

Cancelling the server task is half of it; audio already in the client’s playback buffer keeps playing unless the client clears it, which is why cleared exists.

Where the browser makes it hard

MediaRecorder produces Opus in WebM chunks, and chunks after the first are not decodable on their own — the container header lives in the first one. Decode by feeding every chunk of the connection into one long-lived ffmpeg process’s stdin and reading PCM from its stdout; do not try to decode chunk 5 in isolation.

An AudioWorklet sidesteps this by handing you raw float32 frames to convert and downsample yourself, at more bytes on the wire. For a live transcription product the worklet is the simpler server; for a recording upload MediaRecorder is fine.

Telephony is the reverse problem: the provider’s media stream sends 20 ms frames of base64 μ-law at 8 kHz inside JSON, with its own start / media / mark / stop events, and expects the same shape back. The adapter is a hundred lines and the pipeline behind it does not change.

Operating it

Auth at the handshake, not on the first frame: a token in the query string or a subprotocol, checked before accept() (WebSocket and SSE Authentication).
Heartbeats and timeouts. Ping every 20–30 s; a connection with no audio and no ping for a minute is dead. Idle audio (silence) is still traffic — VAD-gate the STT, not the socket.
Reconnect with a sequence number. Frames carry a counter in the start/resume message so a client that reconnects mid-utterance does not replay five seconds of audio the server already has.
One process per connection, scaled horizontally. Sticky sessions or a connection ID in a shared store; the audio state cannot move between workers mid-stream (Scaling WebSocket Applications).
Test with fixtures. TestClient.websocket_connect, send a known WAV in 640-byte frames with a small sleep, assert the finals. Synthetic tone for load; real recordings for accuracy.

Where it breaks in production

Slow work in the reader. Transcription awaited inline; the socket stops answering pings; the client’s library closes it as dead every 90 seconds under load.
Unbounded buffering. list.append on every frame while the STT vendor is slow; memory grows at 32 KB per second per connection until the worker dies.
Odd-length frames. A proxy re-chunks binary frames, samples split across boundaries, and every frame after the first is misaligned. The remainder buffer is not optional.
Interim transcripts stored as final. “I need to can” and “I need to cancel” both persisted.
TTS that cannot be interrupted. The task is cancelled but the client plays three more sentences from its buffer. Send cleared.
Audio on the event loop thread. Resampling or VAD in numpy on a 20 ms frame is fine; decoding a whole file or running a model is not — asyncio.to_thread or a worker process.

Interview angle 4

“Design live transcription over a WebSocket.” - JSON control frames plus binary audio frames; a start message with rate and encoding; on the server one task reading the socket into a bounded queue and one task feeding the STT stream and sending partial/final transcripts back; a sentinel to flush at end. The bounded queue is the backpressure and the reason a slow vendor never kills the connection.
“The client keeps getting disconnected under load.” - the reader is doing slow work and the socket stops answering pings. Split reading from processing, bound the queue, and move CPU work off the loop. Follow-up: what if memory grows instead? Then the buffer is unbounded; bound it and decide the policy - block the sender, or drop and tell the client.
“How does barge-in work?” - VAD detects the user, an interrupt control message cancels the server’s TTS and LLM tasks, and a cleared message tells the client to flush its playback buffer. The last acknowledged mark says what the user actually heard.
“Binary frames or base64 JSON?” - binary when you own both ends: a third fewer bytes and no decode per frame. Base64 JSON when the peer dictates it - telephony media streams do - or when every frame needs metadata; the rate and encoding still travel once, in start.

Streaming audio over WebSockets

Streaming audio over WebSockets

The shape

The server

Chunk size, latency and the budget

The reply path and barge-in

Where the browser makes it hard

Operating it

Where it breaks in production

Related

Interview angle 4