Streaming LLM responses over SSE

Article.

Every chat product streams tokens from a model provider, through your backend, to a browser. The protocol is Server-Sent Events, covered in Server-Sent Events (SSE): the wire format, EventSource, resume, buffering proxies, idle timeouts, the disconnect chain inside FastAPI. This note is the LLM half, built around one question: “the user clicks Stop. Walk the bytes. Are you still paying?”

The answer needs three things the protocol note does not cover: what your backend does with the provider’s stream (re-emit your own events, never forward theirs), how the disconnect travels one hop further to the provider, and the design choice between a run that dies with the request and one that survives it. Then the parts nobody expects: billing for tokens nobody saw, and token sizes leaking the answer.

provider ─SSE─▶ backend ─SSE─▶ nginx/CDN ─▶ browser
    ▲              │                         │
    └── close ◀────┴──── Stop / tab closed ◀─┘

Re-streaming the provider

The runnable setup below is two FastAPI apps: a fake provider that streams OpenAI Chat Completions-shaped chunks and logs what it generated, and the backend that re-streams it. Start each with uv run --with fastapi --with uvicorn --with httpx uvicorn <file>:app.

The fake provider

# provider.py - a fake LLM streaming Chat Completions-shaped chunks
import asyncio, json, logging
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

log = logging.getLogger("uvicorn.error")
app = FastAPI()
WORDS = "Server-Sent Events are one long HTTP response that never ends early".split()

@app.post("/v1/chat/completions")
async def completions():
    async def gen():
        n = 0
        try:
            for n, word in enumerate(WORDS * 3, 1):     # 33 tokens
                await asyncio.sleep(0.1)                 # decode time per token
                chunk = {"choices": [{"delta": {"content": word + " "}}]}
                yield f"data: {json.dumps(chunk)}\n\n"
            yield "data: [DONE]\n\n"
            log.info("provider: finished, billed %d output tokens", n)
        except asyncio.CancelledError:
            log.info("provider: client gone, stopped at %d tokens", n)
            raise
    return StreamingResponse(gen(), media_type="text/event-stream")

The backend reads it inside the generator

# app.py
import asyncio, json, logging, uuid
import anyio, httpx
from fastapi import FastAPI
from fastapi.sse import EventSourceResponse, ServerSentEvent
from pydantic import BaseModel

log = logging.getLogger("uvicorn.error")
app = FastAPI()
upstream = httpx.AsyncClient(base_url="http://127.0.0.1:9000", timeout=None)

class Prompt(BaseModel):
    prompt: str

async def provider_tokens(prompt: str):
    body = {"stream": True, "messages": [{"role": "user", "content": prompt}]}
    async with upstream.stream("POST", "/v1/chat/completions", json=body) as r:
        async for line in r.aiter_lines():      # an SDK does this parsing for you
            if line.startswith("data: ") and line != "data: [DONE]":
                yield json.loads(line[6:])["choices"][0]["delta"]["content"]

async def save_partial(run_id: str, n: int) -> None:
    await asyncio.sleep(0.05)                   # a database write
    log.info("chat %s: saved %d tokens", run_id, n)

The endpoint

@app.post("/chat", response_class=EventSourceResponse)
async def chat(body: Prompt):
    run_id, sent = uuid.uuid4().hex[:6], 0
    try:
        yield ServerSentEvent(event="run", data={"run_id": run_id}, retry=2000)
        async for tok in provider_tokens(body.prompt):
            sent += 1
            yield ServerSentEvent(event="token", id=str(sent), data={"t": tok})
        yield ServerSentEvent(event="done", id=str(sent + 1), data={})
    except asyncio.CancelledError:
        log.info("chat %s: cancelled after %d tokens", run_id, sent)
        raise
    finally:
        with anyio.CancelScope(shield=True):    # see 01: the finally under cancel
            await save_partial(run_id, sent)

The structural point is that the provider read happens inside the response generator, in its async with. Whatever cancels the generator also closes the provider connection. fastapi.sse (FastAPI 0.135+) frames each event, works for POST, pings after 15 s of silence and sets the no-buffering headers; why the finally needs a shield is in the protocol note.

One event schema of your own

Never forward the provider’s events to the browser. Every provider streams a different shape:

Map them onto a schema you own (run, token, tool_call, usage, error, done), with an id on every event for the resume cursor.

• Raw provider events couple the UI to a vendor, and switching or falling back to another model changes the frontend.
• They can leak tool arguments, internal tool names, or reasoning summaries you did not mean to show.
• Errors after the first byte cannot change the 200. They go in-band as event: error, the way Anthropic sends overloaded_error mid-stream, and the client must handle an answer that stops with an error event.
• Forward usage as a final event, never bill from it: an interrupted stream may not send it at all.

The Stop button, one hop further

The protocol note follows the disconnect from the browser to the generator’s await. In this setup it continues: CancelledError unwinds through async with upstream.stream(...), httpx closes the provider connection, and the provider’s own disconnect handling stops generating. Measured with curl giving up after one second:

$ curl -N --max-time 1 localhost:8000/chat --json '{"prompt":"hi"}'
...
data: {"t": "response "}
id: 7

curl: (28) Operation timed out after 1012 milliseconds with 349 bytes received
app.log       INFO:     chat e247c5: cancelled after 7 tokens
              INFO:     chat e247c5: saved 7 tokens
provider.log  INFO:     provider: client gone, stopped at 8 tokens

Seven tokens delivered, eight generated: whatever is in flight when the cancel lands is still produced.

Real providers add more. Prefill of a long prompt is paid as soon as it starts, and no provider documents exactly what is billed after a disconnect (OpenAI says only that closing the connection cancels a synchronous response).

So measure it: a per-request counter of tokens delivered, reconciled against the provider’s usage report. The metering side is in Worked Design — LLM usage metering.

The detached pump: paying after the user left

A common refactor moves the provider read into a background task that fills a queue, so the response generator only drains the queue:

async def pump():                         # WRONG: a task nobody cancels
    async for tok in provider_tokens(body.prompt):
        await q.put(tok)
    await q.put(None)

tasks.add(asyncio.create_task(pump()))    # kept alive, as the asyncio docs advise
while (tok := await q.get()) is not None:
    yield ServerSentEvent(event="token", data={"t": tok})

$ curl -sN --max-time 1 localhost:8000/chat-detached --json '{"prompt":"hi"}'
(7 token events received)
provider.log  INFO:     provider: finished, billed 33 output tokens

The disconnect cancelled a generator that only read a queue. The task reading the provider had no parent to cancel it, so it generated all 33 tokens for a user who saw 7. Keep the provider read inside the generator, or in a TaskGroup that the generator owns, so cancellation reaches it. This bug is invisible in testing, because every test reads the stream to the end.

Coupled or decoupled runs

The chat() endpoint above is coupled: generation lives and dies with the request. That stops the bill when the user leaves, but a two-second blip on a train is also a disconnect, and then the answer is gone.

A decoupled run generates in its own task, writes every event to a log (a Redis Stream: XADD per event), and readers tail the log from their last id (XREAD BLOCK). Replaying a missed tail costs a read, not a regeneration; the replay endpoint is in the protocol note.

The middle path: a grace period

Generation is decoupled, but abandoned runs are cancelled after a short grace period:

1. Each reader registers on the run; Stop cancels the run task immediately.
2. When the last reader disconnects, schedule loop.call_later(30, cancel_if_abandoned, run), a plain callback, which is safe to register from a cancelled finally.
3. A reader that reconnects within 30 s cancels the timer and resumes from its last id; otherwise the run task is cancelled and billing stops.

Managed versions exist. OpenAI’s background responses keep generating without a connection, resume a dropped stream with starting_after=<sequence_number> and stop with POST /v1/responses/{id}/cancel. LangGraph deployments expose the same choice as on_disconnect (continue or cancel).

Long agent runs that outlast every timeout on the path belong in a job with a status URL (Long-running agent jobs).

Token sizes leak the answer

One event per token gives each TLS record roughly one token’s length, and the sequence of lengths is a fingerprint of the text:

• Weiss et al. (USENIX Security 2024) reconstructed 27% of AI assistant responses from token lengths alone, and inferred the topic of 53%.
• Microsoft’s Whisper Leak (2025-11-07) inferred prompt topics from packet sizes and timing on streamed responses, across many providers.

Providers now pad: OpenAI adds a random obfuscation field to stream deltas by default. A backend that re-emits one event per token rebuilds the leak on the browser leg. Coalesce tokens per frame on a short timer (say 30-50 ms, below what a reader notices), or pad events to size buckets.

Where it breaks in production

• Billed tokens far above delivered tokens: a detached pump, or a decoupled run with no grace-period cancel. Signal: billed minus delivered, per feature.
• The frontend breaks when the model changes: provider events forwarded raw, and the fallback model streams a different shape.
• Answers that silently stop: an error after the first byte with no in-band error event, so the UI waits forever.
• Lost answers on flaky networks: coupled runs with no log to resume from, on mobile clients.
• Cost logs of zero: usage taken from the stream, which an interrupted stream never sent.
• Every token arriving at once in production: a buffering hop, covered with the probe and the fixes in the protocol note.
• Twenty open chats exhaust the database pool: a request-scoped yield dependency held for the whole stream (also in the protocol note).

What to watch

• time to first token at the provider against time to first event at the browser: the gap is your stack;
• open streams and open run tasks, as gauges (len(asyncio.all_tasks()) creeping up means leaked pumps);
• billed minus delivered tokens;
• runs ended by error events, per provider and model.

The trade not taken

WebSocket to the browser. It wins when the client speaks mid-run (voice, interrupting with new input, collaborative editing). For token streaming the traffic is one-way, and WebSocket brings its own reconnect, resume and auth story for no gain. OpenAI’s Responses API WebSocket mode does not change this: it speeds up the backend-to-provider tool loop (OpenAI reports up to about 40% faster on runs with 20 or more tool calls), not the browser leg.

NDJSON over fetch costs little to offer as well (FastAPI streams application/jsonl from a yielding path operation), but has no ids, retry or heartbeat convention; SSE’s framing is worth keeping.

Related

• Server-Sent Events (SSE) - the protocol: wire format, resume, buffering, timeouts, the disconnect chain
• The OpenAI API in practice - OpenAI stream events, background responses
• The Claude Messages API in practice - Anthropic stream events
• Worked Design — LLM usage metering - recording usage from streams that end early
• Pub/sub and Streams - Redis Streams as the run log
• Streaming - streaming agent runs
• Streaming an LLM answer into a React UI - the browser side, the fetch reader loop