Backend / Protocols / Matrix / 03_voip_calls.md

VoIP calls on Matrix

Updated 4 interview angles 6 min read source
On this page8
  1. The split: signalling in the room, media peer to peer
  2. The mechanism: a call is a handful of room events
  3. The homeserver’s part: TURN credentials
  4. Group calls: MatrixRTC and an SFU
  5. Where it breaks in production
  6. The trade not taken: media through your servers by default
  7. Related
  8. Interview angle

VoIP calls on Matrix

Article.

Matrix carries voice and video calls without the homeserver ever touching the audio or video. The question to answer is what the homeserver does in a call, what it deliberately does not do, and what breaks, because a backend engineer on a Matrix product owns the parts that make calls connect (signalling and TURN) and none of the parts that carry them. The call model below is the Matrix spec’s VoIP module; group calls are the newer MatrixRTC work.

The split: signalling in the room, media peer to peer

text 
Alice                                       Bob
  │  signalling: m.call.* room events         │
  ├──► Alice's HS ══ federation ══ Bob's HS ──┤
  │                                           │
  │◄═══ media: WebRTC, direct or via TURN ═══►│
  • Signalling (who calls whom, the SDP offer and answer, ICE candidates) travels as ordinary room events, so it federates, syncs to every device and is stored in room history like a message.
  • Media is WebRTC: SRTP over UDP, directly between the clients when a path exists, through a TURN relay when not (STUN, TURN and coturn).
  • The homeserver’s own job is small: relay the events like any others, and hand clients short-lived TURN credentials.

The mechanism: a call is a handful of room events

json 
{
  "type": "m.call.invite",
  "content": {
    "call_id": "c1760000000123",
    "party_id": "AXJRTDPQ",
    "version": "1",
    "lifetime": 90000,
    "offer": {"type": "offer", "sdp": "v=0\r\no=- 46117 2 IN IP4 127.0.0.1\r\n..."}
  }
}
Event Sent by Means
m.call.invite caller the SDP offer: start a call
m.call.candidates both ICE candidates, in batches
m.call.answer each callee device the SDP answer
m.call.select_answer caller which device’s answer won
m.call.negotiate both renegotiate: add video, share screen
m.call.reject, m.call.hangup either end it

Three details carry most of the design:

  • Every device rings. Bob may have a phone and two laptops; each answers under its own party_id, and the caller’s m.call.select_answer tells the others to stop. Parties are identified by (user_id, party_id).
  • Invites expire. lifetime (at least 90 seconds recommended) plus the event’s age tells a client whether an invite is still live. A phone that was offline syncs an hour of history and must not ring for a call that ended long ago.
  • Glare. If both sides invite at once, the spec resolves it deterministically: a client that receives an invite while preparing its own drops its own; after both are sent, the invite with the lexicographically smaller call_id wins.

Candidates are room events too

Each ICE candidate found after the invite is signalling, and signalling is a room event that is stored and federated. The spec therefore asks clients to send host candidates in the invite, wait briefly (around 200 ms) for STUN and TURN candidates, and batch the rest (about 2 seconds after the invite), ending with an empty candidate:

json 
{
  "type": "m.call.candidates",
  "content": {
    "call_id": "c1760000000123", "party_id": "AXJRTDPQ", "version": "1",
    "candidates": [
      {"candidate": "candidate:1 1 udp 2122260223 192.168.1.23 51720 typ host",
       "sdpMid": "0", "sdpMLineIndex": 0},
      {"candidate": "candidate:2 1 udp 1686052607 <public IP> 58498 typ srflx",
       "sdpMid": "0", "sdpMLineIndex": 0}
    ]
  }
}

Note what that event contains: the device’s private and public IP addresses, written into room history and copied to every server in the room.

The homeserver’s part: TURN credentials

The one VoIP endpoint a homeserver serves:

http 
GET /_matrix/client/v3/voip/turnServer
Authorization: Bearer <access token>

200 OK
{
  "username": "1760003600:@alice:example.org",
  "password": "eSZ5SG+QB+yg...",
  "uris": ["turn:turn.example.org:3478?transport=udp",
           "turn:turn.example.org:3478?transport=tcp",
           "turns:turn.example.org:443?transport=tcp"],
  "ttl": 3600
}

Synapse mints these from a secret it shares with coturn, so neither keeps a user table:

yaml 
# homeserver.yaml
turn_uris:
  - "turn:turn.example.org:3478?transport=udp"
  - "turn:turn.example.org:3478?transport=tcp"
  - "turns:turn.example.org:443?transport=tcp"
turn_shared_secret: "<the same value as coturn's static-auth-secret>"
turn_user_lifetime: 1h
turn_allow_guests: false

The username is an expiry timestamp and the user ID; the password is an HMAC-SHA1 of it with the shared secret, which coturn recomputes. Rate-limit the endpoint like any credential issuer.

Group calls: MatrixRTC and an SFU

The 1:1 model does not scale to groups. In a mesh, each participant sends its stream to every other participant:

text 
participants   streams each uploads (mesh)   with an SFU
     2                    1                       1
     4                    3                       1
     8                    7                       1

At 1.5 Mbit/s per video stream, an eight-person mesh asks every phone for 10.5 Mbit/s of upload.

Element Call therefore uses MatrixRTC: call membership lives in room state, and media goes through a LiveKit SFU (selective forwarding unit) that receives one stream from each participant and forwards it to the others.

A small service, lk-jwt-service, exchanges a Matrix OpenID token for a LiveKit access token, so only room members reach the SFU. MatrixRTC is still moving through MSCs (how a homeserver advertises its SFU has changed between Element Call releases), so check the current Element Call and Synapse release notes before deploying it.

Where it breaks in production

  • Calls fail for some users only. Those behind symmetric NAT or on mobile networks, when no TURN server is configured or it is reachable only on UDP 3478. The signalling succeeds, so it looks like a client bug.
  • Credentials rejected. turn_shared_secret differs from coturn’s static-auth-secret, or coturn’s clock is wrong: the expiry in the username is checked against coturn’s clock, so a skewed server rejects fresh credentials or accepts expired ones.
  • IP addresses in room history. Host and server-reflexive candidates are persisted and federated. Calls belong in 1:1 rooms (the spec’s own security note: in a group room another member can intercept and answer), and a privacy-sensitive client can force relay-only ICE, at the cost of relaying every call.
  • Ghost ringing. A client that ignores lifetime and event age rings for stale invites after a long sync.
  • Group calls without SFU capacity. The SFU’s UDP port range and bandwidth are a separate service to size and secure, beside coturn.

The trade not taken: media through your servers by default

A tempting design for control and recording is to route all media through your own servers (an SFU even for 1:1 calls, or relay-only ICE). It makes every call cost server bandwidth and adds a hop of latency to calls that could have gone direct, and with end-to-end encrypted media the server cannot record it anyway.

Matrix’s split (signalling through the homeserver, media direct when possible, TURN when not, an SFU only for groups) keeps the homeserver cheap and the media path short.

Interview angle 4

  • “What does the homeserver do in a Matrix call?” - carries the signalling as room events (invite, candidates, answer, hangup) and hands out time-limited TURN credentials. It never carries media.
    • Follow-up: Bob has three devices. Which one answers? All of them ring; each answers with its own party_id, and the caller’s m.call.select_answer picks one and stops the rest.
  • “Why do calls fail for some users only?” - symmetric NAT and strict networks need a TURN relay; without one, or with only UDP 3478 reachable, ICE finds no path while signalling works fine.
    • Follow-up: and if TURN is configured but rejects them? A shared-secret mismatch between Synapse and coturn, or a skewed clock on coturn, since the credential’s expiry is checked against it.
  • “What is the privacy cost of in-room signalling?” - candidates carry the device’s IP addresses into room history on every server in the room. Keep calls to 1:1 rooms, or force relay-only ICE.
  • “How do group calls work?” - a mesh needs N-1 uploads per person, seven streams at eight people, so Element Call uses MatrixRTC with a LiveKit SFU; a JWT service turns a Matrix OpenID token into SFU access.
Previous
Sygnal: the Matrix push gateway