AI & ML / Speech & realtime / 01_speech_stt_tts.md

Speech: STT and TTS

Updated 5 interview angles 5 min read source
On this page5
  1. Speech to text
  2. Text to speech
  3. Cost and architecture
  4. Testing a speech pipeline
  5. Interview angle

Speech: STT and TTS

The audio layer under voice AI products. One of the roles in scope names “AI solutions for communication, audio/video streaming, chatbots”, which is this plus Realtime voice agents.

Speech to text

Modern ASR is a sequence model over audio features. What matters in an interview is the operational shape, not the architecture.

Batch versus streaming. Batch transcription sends a complete file and returns the full transcript — higher accuracy, because the model sees the whole context, and unsuitable for conversation. Streaming emits partial hypotheses as audio arrives, revising them as more context appears. The revision behaviour is the part that surprises people: an interim transcript can change before it is final, so downstream logic must key on final results.

python 
async for event in stt.stream(audio):
    if event.is_final:
        await handle(event.text)   # act here
    else:
        ui.show_draft(event.text)  # display only

Appending every interim to a transcript is the classic bug: “I need to can” becomes “I need to cancel” and you have stored both.

Metrics. Word Error Rate (substitutions + insertions + deletions, over reference words) is the standard, and it is a blunt instrument. WER weights every word equally, so getting “not” wrong scores the same as getting “the” wrong. For a product, measure what matters: entity accuracy on names, numbers and drug names; intent accuracy after the transcript reaches the LLM.

python 
# The metric that reflects the product.
def entity_accuracy(cases):
    hit = sum(
        e in c.transcript
        for c in cases for e in c.critical_entities
    )
    total = sum(len(c.critical_entities) for c in cases)
    return hit / total

A model at 12% WER that never misses a drug name beats one at 8% that sometimes does. Report both, and let the product metric decide.

Where accuracy actually degrades: accents and dialects, domain vocabulary, overlapping speakers, telephone-band audio (8 kHz narrowband loses a lot), and background noise. Vendor WER figures are quoted on clean read speech and do not survive a call centre.

Features you will need to name:

Feature For
Diarization who spoke when — required for multi-party transcripts
Word-level timestamps aligning transcript to audio, highlighting, editing
Custom vocabulary / biasing product names, drug names, SKUs the base model has never seen
Punctuation and formatting raw ASR output is an unpunctuated stream
VAD (voice activity detection) segmenting speech from silence; the input to turn detection
Language identification multilingual deployments

Custom vocabulary is the highest-leverage fix in most real deployments. A domain term transcribed wrong every time destroys downstream extraction, and biasing costs far less than fine-tuning.

Options: hosted APIs from the major cloud and speech vendors, or self-hosted open-weights models (the Whisper family and its faster reimplementations) when data cannot leave your boundary or volume makes per-minute pricing painful. Self-hosting is genuinely viable here — unlike frontier LLMs, good ASR runs on modest GPUs.

Text to speech

Quality axes: naturalness, latency, controllability (pacing, emphasis, pronunciation via SSML or phoneme hints), and voice cloning. Cloning carries consent and likeness obligations — the EU AI Act’s transparency rules require disclosure when a user is interacting with synthetic media, in force since 2 August 2026. See Guardrails and safety.

Streaming TTS is the requirement for conversation. Synthesising a whole reply before playback adds its full duration to perceived latency. Stream audio chunks as they are generated, and start synthesis on the first sentence rather than waiting for the LLM to finish.

Time to first byte is the metric that matters, not total synthesis time. A voice that starts in 150 ms and finishes slowly feels responsive; one that starts in 900 ms does not, regardless of throughput.

Cost and architecture

Audio is billed per minute or per character, not per token, which changes the optimisation:

  • Cache TTS for anything repeated — prompts, menu options, confirmations. A surprising share of a voice product’s speech is fixed strings.
  • Do not transcribe silence. VAD-gate the stream before it reaches the ASR.
  • Choose the transport deliberately: WebRTC for live conversation, WebSocket for server-to-server streaming, plain HTTP for batch. See WebSockets.
  • Store audio only if you need it, and if you do, treat it as personal data with a retention policy. Voice is biometric-adjacent under GDPR and recordings in a regulated setting carry the same obligations as any other record. See PHI, privacy, and secure coding in regulated systems.

Testing a speech pipeline

  • A held-out audio set with reference transcripts, including the hard cases — accents, noise, domain vocabulary — not just clean samples.
  • Track WER and the downstream metric, because the two diverge. A model with worse WER that gets entity names right is the better product choice.
  • Regression-test on every model or prompt change; providers update models under the same endpoint name, which silently changes your accuracy.
  • Synthetic audio for load testing, real audio for accuracy.

Interview angle 5

  • “Streaming or batch transcription?” - streaming for conversation, accepting lower accuracy and interim results that get revised; batch when the whole file is available and accuracy matters more than latency. The gotcha is that interim transcripts change, so downstream logic must act on finals.
  • “Why is WER a poor product metric?” - it weights every word equally. Getting a negation or a drug name wrong is catastrophic; getting an article wrong is not. Measure entity-level accuracy and the downstream task metric alongside it.
  • “Transcription quality is poor on our domain terms - what do you do first?” - custom vocabulary or keyword biasing. It is cheap, immediate, and fixes the specific failure. Fine-tuning is a much larger commitment for a problem biasing usually solves.
  • “What latency metric matters for TTS?” - time to first audio byte, not total synthesis time. Stream chunks and start synthesising on the first sentence rather than waiting for the full LLM response.
  • “Would you self-host speech models?” - plausibly yes, unlike frontier LLMs. Good open-weights ASR runs on modest hardware, and self-hosting answers both the data-residency question and the per-minute cost at volume.