Voice platforms and cloning
Speech: STT and TTS covers the mechanics and Realtime voice agents the architecture. This is the landscape: who provides what, and the consent problem that comes with synthetic voice.
The providers, by what they are for
| Provider | Strength |
|---|---|
| ElevenLabs | synthesis quality, cloning, many languages |
| Deepgram | fast streaming ASR, low cost at volume |
| AssemblyAI | ASR plus diarization and summarisation |
| Cartesia | very low latency synthesis |
| Cloud STT/TTS | already in your tenancy, compliance |
| Whisper (open) | self-hosted, no per-minute cost |
ElevenLabs is the name to know for synthesis — it is the reference point for quality and the one ads mention. Deepgram and AssemblyAI are the ASR counterparts, and the hyperscaler services (Azure Speech, Google, Amazon Transcribe/Polly) win when the data must stay inside an existing boundary.
The frontier providers now offer speech natively too, which collapses the cascade into one call — see the architecture comparison in Realtime voice agents.
Choosing, in the order that matters
- Latency, if it is conversational. Time to first audio byte decides whether the interaction feels like a conversation or a form.
- Data boundary, if it is regulated. Voice is biometric data in several jurisdictions, which narrows the list fast.
- Language coverage, checked on your accents rather than the marketing list — quality varies enormously within a claimed language.
- Cost per minute, which is the unit that scales, not per token.
Gotcha: benchmark on your own audio. Published word error rates come from clean read speech; your traffic is a call centre with crosstalk, hold music and a phone codec. The gap between the two is routinely a factor of several.
Self-host or buy
| Hosted API | Self-hosted | |
|---|---|---|
| Time to ship | hours | weeks |
| Cost at volume | per minute | GPU, fixed |
| Data boundary | vendor’s | yours |
| Quality | best available | good, lags |
The crossover is real but later than people expect: a GPU running Whisper costs the same whether you send it one hour of audio or a thousand, so self-hosting wins at sustained volume and loses at spiky, low volume.
Compliance is the other reason, and it is often the deciding one. If voice recordings cannot leave your infrastructure, the hosted option is not on the list regardless of price.
Voice cloning, and the part that matters
Modern systems clone a voice from a short sample — seconds, not hours. That capability is what makes the technology commercially interesting and legally fraught.
The rules that keep you out of trouble:
- Consent must be explicit and from the voice’s owner. Having a recording is not permission to synthesise from it.
- Disclose synthetic audio. The EU AI Act’s Article 50 transparency obligations apply to synthetic media and are in force as of 2026-08 — see Model governance and responsible AI.
- Voice is biometric data under GDPR and similar regimes, so retention, minimisation and erasure rules apply to the samples as well as the output.
The threat model is worth stating unprompted: cloned voice defeats voice-based authentication, and it powers a live category of fraud where a familiar voice asks for an urgent transfer. If a product uses voice for identity, that is now a design flaw rather than a feature.
What a production pipeline looks like
audio in ──▶ ASR ──▶ redact PII ──▶ agent
│
audio out ◀── TTS ◀── guardrails ◀───┘Two boxes people leave out:
Redaction before the transcript is stored. A call transcript contains card numbers and personal detail spoken aloud, and it lands in your logs by default.
async for event in stt.stream(audio):
if not event.is_final:
continue
clean, found = redact(event.text)
# never event.text
await store.append(call_id, clean)
await agent.send(clean)The ordering is the whole control: redact between the transcript and both sinks. Redacting only on the way to the model still leaves the raw card number in your database.
See PII, privacy and the EU AI Act.
A recording and consent path. Whether the call is recorded, who was told, and how long it is kept are product decisions with legal consequences, not infrastructure defaults.
Related
Interview angle 6
- “Which voice providers would you use?” - ElevenLabs is the reference for synthesis quality and cloning; Deepgram or AssemblyAI for streaming ASR; the hyperscaler services when the data must stay in an existing tenancy; self-hosted Whisper at sustained volume.
- “How would you choose?” - latency first if it is conversational, then the data boundary if it is regulated, then language coverage tested on your own accents, then cost per minute. Published word error rates come from clean read speech and do not survive a call centre.
- “Self-host or buy?” - a GPU costs the same at one hour or a thousand, so self-hosting wins at sustained volume and loses at spiky low volume. Compliance often decides it before cost does: if recordings cannot leave your infrastructure, hosted is not on the list.
- “What are the rules around voice cloning?” - explicit consent from the voice’s owner, disclosure of synthetic audio under the EU AI Act’s Article 50 obligations, and treating voice as biometric data for retention and erasure. Possessing a recording is not permission to synthesise from it.
- “What’s the security implication?” - cloned voice defeats voice-based authentication, and a familiar voice requesting an urgent transfer is a live fraud pattern. Voice as an identity factor is now a design flaw.
- “What do people leave out of the pipeline?” - PII redaction before transcripts are stored, since card numbers get spoken aloud and land in logs by default, and an explicit recording-and-consent path.