Audio fundamentals for backend engineers

“Capturing, converting and transmitting audio streams” is a bytes problem before it is a speech problem. A backend engineer on a voice product spends most of the audio time on sample rates, byte alignment and codecs, and the bugs are all the same three: the wrong rate, the wrong layout, the wrong container.

This note is the arithmetic and the tools; Streaming audio over WebSockets is the pipeline.

PCM: what a second of audio is

Raw audio is PCM: a sequence of samples, each the amplitude at one instant. Three numbers define it, and every stage must agree on all three:

RATE, WIDTH, CHANNELS = 16_000, 2, 1        # 16 kHz, int16, mono
bytes_per_second = RATE * WIDTH * CHANNELS  # 32,000
frame_ms = 20
frame_bytes = RATE * frame_ms // 1000 * WIDTH * CHANNELS   # 640
# 20 ms at 16 kHz mono int16 = 320 samples = 640 bytes

Speech models want 16 kHz mono int16, little-endian (s16le). A browser microphone gives 48 kHz float32; a phone line gives 8 kHz μ-law. Everything in between is conversion, and the conversion is where the quality goes.

The three bugs

• Rate mismatch. Audio recorded at 48 kHz and labelled 16 kHz plays three times too slow and low; the reverse is chipmunk speech. A transcription model given the wrong rate returns garbage with high confidence. The rate is metadata that travels with the bytes — put it in the stream’s start message, and assert it at every boundary.
• Layout mismatch. Stereo interleaved samples read as mono are two voices at double speed; float32 read as int16 is noise. int16 from float needs scaling by 32767 and clipping first, or a loud frame wraps around into a burst of static.
• Odd-length chunks. A network chunk that ends mid-sample splits an int16 in two; the next chunk starts on the wrong byte and every sample after is corrupt. Keep a remainder buffer and only decode whole samples.

import numpy as np

class PcmBuffer:
    """Accumulate bytes; hand out whole int16 frames; keep the odd byte."""
    def __init__(self, frame_bytes: int):
        self.buf = bytearray()
        self.frame_bytes = frame_bytes

    def push(self, chunk: bytes):
        self.buf += chunk
        while len(self.buf) >= self.frame_bytes:
            frame = bytes(self.buf[:self.frame_bytes])
            del self.buf[:self.frame_bytes]
            yield np.frombuffer(frame, dtype="<i2")     # little-endian int16

def float_to_int16(x: np.ndarray) -> np.ndarray:
    return (np.clip(x, -1.0, 1.0) * 32767).astype(np.int16)

Containers and codecs

The distinction to keep straight: a codec compresses samples (Opus, MP3, μ-law); a container wraps encoded frames with headers and timing (WebM, Ogg, WAV, MP4). A “raw stream” has neither — which is what a speech model wants and what a browser will not give you without work. μ-law is the odd one: one byte per sample, logarithmic, 8 kHz, and still what a phone call is. Decoding it is a table lookup:

def ulaw_to_pcm16(b: bytes) -> np.ndarray:
    """G.711 mu-law -> int16. Matches audioop.ulaw2lin(b, 2)."""
    u = ~np.frombuffer(b, dtype=np.uint8)
    sign, exponent, mantissa = u & 0x80, (u >> 4) & 0x07, u & 0x0F
    mag = (((mantissa.astype(np.int32) << 3) + 0x84) << exponent) - 0x84
    return np.where(sign, -mag, mag).astype(np.int16)

The stdlib audioop module that used to do this was removed in Python 3.13; audioop-lts on PyPI restores it, and the numpy version above is what you write when you would rather not depend on it.

Converting

ffmpeg is the universal converter, and the way to use it from a service is a subprocess with pipes, not temp files:

import subprocess

def decode_to_pcm16k(encoded: bytes) -> bytes:
    """Any container/codec ffmpeg knows -> raw s16le mono 16 kHz."""
    cmd = ["ffmpeg", "-loglevel", "error", "-i", "pipe:0",
           "-f", "s16le", "-acodec", "pcm_s16le", "-ac", "1", "-ar", "16000",
           "pipe:1"]
    return subprocess.run(cmd, input=encoded, capture_output=True,
                          check=True).stdout

For a live stream keep one ffmpeg process per connection with stdin/stdout pipes and feed chunks as they arrive; pydub wraps the same binary for whole files.

Resampling in-process is soxr (soxr.resample(x, 48000, 16000)) or scipy.signal.resample_poly; librosa and torchaudio do it too with heavier dependencies. Reading and writing WAV/FLAC is soundfile (sf.read(path, dtype="int16")). Naive decimation — taking every third sample — aliases; use a resampler.

Capture

• Browser. getUserMedia then either MediaRecorder (Opus in WebM chunks, compact, needs ffmpeg to decode) or an AudioWorklet (raw float32 frames at the context rate, convert and downsample in the worklet or on the server). For live transcription, raw PCM from a worklet is simpler to consume; MediaRecorder chunks are cheaper on the wire but the chunks after the first are not decodable on their own — see the pipeline note.
• Telephony. Twilio-style media streams deliver 20 ms μ-law frames as base64 JSON over a WebSocket; decode with the function above.
• Server-side microphones (kiosks, devices): sounddevice (PortAudio) with a callback per block; pyaudio is the older equivalent.

VAD and silence

Voice activity detection decides which frames carry speech. It gates the transcription (do not pay to transcribe silence), it is the input to endpointing (a run of silence ends the utterance), and it is what makes turn-taking possible (Realtime voice agents).

webrtcvad is the classic — 10/20/30 ms int16 mono frames at 8/16/32/48 kHz, aggressiveness 0–3, fast and CPU-only, weak on noise. Silero VAD is the modern default: a small neural model, far better on noisy audio, still cheap enough to run per frame on CPU. An energy threshold is the fallback that works until the first air-conditioner.

Where it breaks in production

• The header in the stream. A WAV file’s 44 bytes fed to the model as audio: a click, then correct audio 22 samples late. Strip containers; send raw PCM.
• Float clipping. A loud speaker, float samples above 1.0 cast to int16 without clipping, and the wrap-around is a burst of noise the model transcribes as words.
• ffmpeg not flushed. A live decode that never sends EOF holds the last frames in ffmpeg’s buffer; the final word of every utterance vanishes.
• Narrowband into a wideband model. 8 kHz telephony upsampled to 16 kHz is still 8 kHz of information; accuracy is measurably lower and the fix is a model or config tuned for telephony, not a resampler.
• Audio in logs and on disk. Recordings are personal data with a retention policy; a debug dump of raw frames is a breach.

Related

• Streaming audio over WebSockets - the pipeline these bytes flow through.
• Speech: STT and TTS - what the model does with them.
• Realtime voice agents - VAD, endpointing and barge-in in a conversation.
• TCP vs UDP - why WebRTC carries audio over UDP.