Server-Sent Events (SSE)

Article.

Server-Sent Events is one long HTTP response that the server keeps writing to, in a line-based text/event-stream format the browser parses as it arrives. It is one-way, server to client, and it is plain HTTP: cookies, auth headers, load balancers and HTTP/2 multiplexing work unchanged, and the browser’s EventSource reconnects and resumes on its own. Notifications, live dashboards, job progress and LLM token streams all fit.

source ─▶ FastAPI ─▶ nginx ─▶ CDN ─▶ browser
   ▲         │                         │
   └─ stop ◀─┴────── disconnect ◀──────┘

Four properties must hold on that path, and most of this note is one of them failing. Live: every hop forwards an event the moment it arrives. Alive: no hop kills a stream for being quiet. Resumable: a dropped connection continues from the last event. Released: when the client leaves, the server stops the work it was doing for it.

The wire format, byte by byte

A token stream from the FastAPI endpoint shown further down, as curl prints it; --raw keeps the HTTP/1.1 chunk framing that curl normally decodes away:

$ curl -iN --raw localhost:8000/chat --json '{"prompt":"x"}'
HTTP/1.1 200 OK
content-type: text/event-stream; charset=utf-8
cache-control: no-cache
x-accel-buffering: no
transfer-encoding: chunked

33
event: run
data: {"run_id": "963010"}
retry: 2000


30
event: token
data: {"t": "Server-Sent "}
id: 1

Each line ends in LF, and a blank line ends an event. 33 is a chunk length in hex (51 bytes), so each yield left the server as exactly one chunk. HTTP/2 carries the same bytes in DATA frames instead. The browser parses one continuous text stream either way, so a proxy may split or merge chunks; holding them back is what breaks streaming.

Headers and fields

Only Content-Type: text/event-stream is required. Cache-Control: no-cache stops a cache replaying a stored copy, and X-Accel-Buffering: no turns off nginx buffering for this response. Skip Connection: keep-alive: HTTP/1.1 defaults to it, and HTTP/2 forbids connection-specific headers (RFC 9113 section 8.2.2).

How the browser parses the stream

The WHATWG HTML standard specifies the parser exactly, and it is short enough to be the mechanism itself. Here it is in Python, minus the EventSource object:

import json, re

def parse_sse(text: str):
    """WHATWG event-stream parsing. Each event is (type, data, last_id)."""
    events, data, etype, last_id, retry = [], "", "", "", None
    *lines, _unterminated = re.split(r"\r\n|\r|\n", text)
    for line in lines:
        if line == "":                                  # blank line: dispatch
            if data:
                events.append((etype or "message", data[:-1], last_id))
            data = etype = ""
        elif line.startswith(":"):                      # comment: heartbeat
            continue
        else:
            field, _, value = line.partition(":")
            value = value.removeprefix(" ")             # ONE space, not all
            if field == "data":
                data += value + "\n"
            elif field == "event":
                etype = value
            elif field == "id" and "\0" not in value:
                last_id = value
            elif field == "retry" and value.isascii() and value.isdigit():
                retry = int(value)
    return events, retry                                # unfinished event: gone

The awkward cases

Feed it every rule at once:

stream = (
    ": ping\r\n"                    # comment
    "retry: 2500\n"
    "event: token\n"
    "data:  two spaces\n"           # only one leading space is stripped
    "data: second line\r\n"         # CRLF, LF and CR all end a line
    "id: 7\n\n"
    'data: {"t": "x"}\r\r'
    "id: 8\0\n"                     # NUL inside an id: the field is ignored
    "data: next\n\n"
    "retry: 3s\n"                   # not all digits: ignored
    "data: cut off by a proxy\n"    # no blank line before the stream ends
)
events, retry = parse_sse(stream)
print(*events, f"retry={retry}", sep="\n")

('token', ' two spaces\nsecond line', '7')
('message', '{"t": "x"}', '7')
('message', 'next', '7')
retry=2500

The last event never fires: without its closing blank line it is dropped when the stream ends, so a proxy that cuts a stream mid-event loses the final events with no error anywhere. End every stream with an explicit done event, the only reliable way a client can tell “finished” from “cut”. Two rules the sample skips: the stream must be UTF-8, and an empty id: resets the id.

The blank-line injection

Payloads are full of newlines (markdown from a model, a log line, user text), and a hand-rolled frame passes them through:

token = "Step 1\n\nStep 2"
parse_sse(f"data: {token}\n\n")[0]                # [('message', 'Step 1', '')]
parse_sse(f"data: {json.dumps(token)}\n\n")[0]    # one event, whole string intact

Step 2 raises nothing: it parses as an unknown field named Step 2 and is ignored. JSON-encode the payload, which escapes the newlines, or send one data: line per line of text. fastapi.sse always JSON-encodes data=.

EventSource: reconnect, resume, and when it gives up

The browser’s built-in client earns its keep after a failure. Per the WHATWG spec and the engines’ source:

• It reconnects after a network error, and also after a normal end of the response, waiting the retry delay: 3 s by default in Chromium and WebKit, 5 s in Firefox, which floors retry at 500 ms. The delay is fixed; the spec permits exponential backoff, and no major engine implements it.
• It resumes. The reconnect sends Last-Event-ID with the last id seen. A new EventSource, after a page reload for instance, has none to send.
• It gives up for good on any status other than 200, or a Content-Type other than text/event-stream: readyState becomes CLOSED, with no retry. A 204 is the documented stop, and a 502 from a load balancer mid-deploy is the accidental one.

So a GET whose handler starts work starts it again on every reconnect, forever, unless the client calls close() on done or the server answers 204. And an error event with readyState === EventSource.CLOSED means you build a new EventSource yourself, with your own backoff.

Six connections per origin

Over HTTP/1.1 a browser opens at most six connections per origin, shared by every tab, and each open EventSource holds one; a seventh stream and every other fetch to that origin queue behind them (Chrome and Firefox: won’t fix, per MDN). Over HTTP/2 streams multiplex on one connection, bounded by SETTINGS_MAX_CONCURRENT_STREAMS (nginx defaults to 128). Terminate HTTP/2 at the edge, or share one stream across tabs with a SharedWorker.

EventSource or fetch

EventSource can only GET, with no body and no custom headers, and a chat turn is a POST carrying a JSON prompt and usually an Authorization header. So chat UIs read the stream with fetch() and parse it themselves (the reader loop is in the frontend note).

Decode with pipeThrough(new TextDecoderStream()) and split on all three line endings, or use @microsoft/fetch-event-source.

With fetch you give up everything in the previous section: no reconnect, no Last-Event-ID. Rebuild both with the last id seen, a backoff loop and an ?after=<id> cursor, and refresh the token before reconnecting, or a 401 silently ends the stream. A POST that starts a run plus a GET that streams it gets EventSource back for resume when auth is a cookie (WebSocket and SSE Authentication).

Serving SSE from FastAPI

FastAPI 0.135.0 added fastapi.sse. Put response_class=EventSourceResponse on a path operation that yields, and each ServerSentEvent(data, raw_data, event, id, retry, comment) is framed for you, for any method including POST. It sends a ping comment after 15 s without an event and sets Cache-Control: no-cache and X-Accel-Buffering: no. On older versions, sse-starlette’s EventSourceResponse does the same job.

A stream with ids and a log

import uuid
from collections.abc import AsyncIterable
from typing import Annotated

from fastapi import Depends, FastAPI, Header, HTTPException
from fastapi.sse import EventSourceResponse, ServerSentEvent
from pydantic import BaseModel

app = FastAPI()
LOGS: dict[str, list[ServerSentEvent]] = {}   # stand-in for a Redis Stream per run

class Prompt(BaseModel):
    prompt: str

@app.post("/chat", response_class=EventSourceResponse)
async def chat(body: Prompt) -> AsyncIterable[ServerSentEvent]:
    run_id = uuid.uuid4().hex[:6]
    log = LOGS[run_id] = []
    yield ServerSentEvent(event="run", data={"run_id": run_id}, retry=2000)
    async for tok in tokens(body.prompt):     # any async source: a model, a queue
        ev = ServerSentEvent(event="token", id=str(len(log) + 1), data={"t": tok})
        log.append(ev)
        yield ev
    log.append(ServerSentEvent(event="done", id=str(len(log) + 1), data={}))
    yield log[-1]

Every event carries an id and goes into a log first: that is what makes the stream resumable. Errors after the first byte cannot change the 200, so they go in-band as event: error, followed by the end of the stream.

Resume: answering Last-Event-ID

Resume is the server’s half of the deal: keep the events, and answer Last-Event-ID from the log rather than by redoing the work.

def cursor(
    run_id: str,
    last_event_id: Annotated[str | None, Header()] = None,
    after: int = 0,                       # fetch() clients send no Last-Event-ID
) -> int:
    if run_id not in LOGS:
        raise HTTPException(status_code=404)
    start, events = int(last_event_id or after), LOGS[run_id]
    if events and start >= len(events) and events[-1].event == "done":
        raise HTTPException(status_code=204)      # finished: stop reconnecting
    return start

@app.get("/runs/{run_id}/events", response_class=EventSourceResponse)
async def replay(run_id: str, start: Annotated[int, Depends(cursor)]):
    for ev in LOGS[run_id][start:]:       # ids are 1-based positions in the log
        yield ev

The 204 has to come from a dependency, which runs before the response starts. Raised inside the generator it is too late: the 200 and its headers are gone, the stream simply ends, and EventSource reconnects to it forever.

This replay ends where the log ends. In production the log is a Redis Stream: XADD per event, and XREAD BLOCK from the client’s last id returns the missed tail and then blocks for the next event, so one loop covers replay and live. Resume also needs a producer that outlives the connection; if the work is cancelled on disconnect (next section), there is nothing left to resume.

When the client leaves

1. Closing the tab or AbortController.abort() closes the connection; on HTTP/2 the browser resets just that stream.
2. Each proxy closes its upstream in turn. nginx does by default (proxy_ignore_client_abort off).
3. uvicorn’s connection_lost marks the request disconnected, so the next receive() returns http.disconnect.
4. uvicorn 0.54 reports ASGI HTTP spec 2.3, so Starlette’s StreamingResponse runs listen_for_disconnect beside the body loop and cancels the task group.
5. CancelledError lands at the generator’s current await, and its async with blocks unwind, closing what they hold: an upstream HTTP stream, a database cursor, a Redis subscription.

Step 4 depends on the server. One reporting ASGI spec 2.4 or later gets no listener, and Starlette learns only when a send() fails (OSError, re-raised as ClientDisconnect). A generator stuck in a minute-long wait then holds its upstream for that minute; fastapi.sse bounds it at 15 s, since its ping is a send. request.is_disconnected() is a non-blocking receive(): a fallback for loops that never yield, not the mechanism.

The finally block runs under cancellation

AnyIO cancellation is level-triggered: inside a cancelled scope every await raises again, including the one in the cleanup that saves the partial result.

# WRONG: cancelled at its first await. No error, no log line, no row.
finally:
    await save_partial(run_id, sent)

# RIGHT: shield the bounded cleanup from the scope's cancellation
finally:
    with anyio.CancelScope(shield=True):
        await save_partial(run_id, sent)

Shield only bounded work; a shielded call to a hung database hangs the request, and work that must survive a crash belongs in an outbox. A hand-rolled StreamingResponse adds one more trap: if the cancel lands while Starlette awaits send() (a slow client, a full buffer), the generator is parked at yield and closes only when garbage-collected.

Where it breaks in production

The burst: every event arrives at the end

It streams on localhost and arrives in production as nothing, then everything. Bisect with a probe at each hop in order (app port, proxy, load balancer, CDN): the first hop where chunks merge is the culprit.

# probe.py URL [--gzip]: when does each chunk arrive, and how big is it?
import sys, time
import httpx

url, gzip = sys.argv[1], "--gzip" in sys.argv
headers = {"accept-encoding": "gzip" if gzip else "identity"}
t0 = time.perf_counter()
with httpx.stream("POST", url, json={"prompt": "hi"}, headers=headers,
                  timeout=None) as r:
    print(r.status_code, r.headers.get("content-encoding", "identity"))
    for chunk in r.iter_raw():            # raw bytes as they arrive
        print(f"+{time.perf_counter() - t0:5.2f}s {len(chunk):5d} B")

What buffers, and the fix

The usual buffers are compression and proxy buffering. gzip on with text/event-stream in gzip_types holds the stream until the compressor has enough to emit, often the whole answer in one chunk at the end. In nginx, proxy buffering is per response (X-Accel-Buffering: no from the app) or per location:

location /api/chat {
    proxy_pass http://app;
    proxy_http_version 1.1;          # the default only since nginx 1.29.7
    proxy_set_header Connection "";  # upstream keepalive; default since 1.29.7 too
    proxy_buffering off;             # or the app's X-Accel-Buffering: no
    proxy_read_timeout 1h;           # longest silence allowed between two reads
    gzip off;
}

The same bisection finds the other buffers. API Gateway REST APIs buffer unless responseTransferMode is STREAM (since November 2025), Cloudflare Tunnel buffers anything that is not text/event-stream, and a CDN compression rule that matches the type holds it the way gzip does. Starlette’s GZipMiddleware skips text/event-stream, so it is not to blame.

Idle timeouts cut quiet streams

A stream is quietest exactly when it matters: a long prefill, a tool call, a slow query before the first event. fastapi.sse pings after 15 s of silence, under every limit above and in line with the WHATWG advice of a comment “every 15 seconds or so”. A hand-rolled stream needs its own : ping on a timer.

Sync generators exhaust the thread pool

FastAPI iterates a def generator with iterate_in_threadpool, and each next() holds one of AnyIO’s 40 default threads while it blocks (FastAPI: sync code in an async route, run_in_threadpool). Forty streams blocked on a sync client leave every sync endpoint and sync dependency in the app queued, health checks included. An async stream costs a socket and a coroutine, so stream from async code.

A yield dependency holds its connection for the whole stream

Since FastAPI 0.118.0 yield-dependency teardown runs after the response is sent, which for a stream means after the last event. From 0.106.0 to 0.117 it ran before streaming, the source of the old “session closed mid-stream” bug. On SQLAlchemy’s default pool (5 plus 10 overflow, 30 s pool_timeout) the sixteenth concurrent long stream makes every database-backed endpoint wait 30 s, then raise TimeoutError.

Depends(get_db, scope="function") (0.121.0) releases the connection before the generator’s first line, so the stream cannot use it at all. Load what the stream needs up front, and open a short session inside the generator to persist results (Dependencies and injection).

Deploys

uvicorn waits for open connections at shutdown (timeout_graceful_shutdown defaults to None), and Kubernetes sends SIGKILL after terminationGracePeriodSeconds, 30 s by default, so a rollout cuts every long stream. Drain on SIGTERM: end each stream cleanly, which EventSource treats as a cue to reconnect, with a final event that tells fetch clients the same. They resume from their last id on a new pod, if the log lives in Redis.

What to watch

• open streams, as a gauge
• time to first event at the browser against time to first event at the source
• inter-event gaps as a histogram; a burst is one long gap and then zeros
• 504 and 524 rates per hop
• len(asyncio.all_tasks()) creeping up: leaked generators or tasks

The trade not taken

WebSocket loses on one-way traffic because it brings its own reconnect, resume and auth story for no gain (The WebSocket protocol). NDJSON costs little to offer, since FastAPI 0.134.0 streams application/jsonl from any yielding path operation.

Runs measured in minutes want a job and a status URL; polling is in Short Polling vs Long Polling.

Related

• Streaming LLM responses over SSE - the LLM half: re-streaming a provider, Stop and billing, coupled vs decoupled runs
• Streaming an LLM answer into a React UI - the browser consumer: EventSource and the fetch reader loop
• Race conditions and AbortController - the Stop button and AbortController
• WebSocket and SSE Authentication - auth when the browser cannot send headers
• WebSocket vs Alternatives - SSE, WebSocket and polling side by side
• Short Polling vs Long Polling - the HTTP-only alternatives, with cursors and proxy timeouts
• Nginx WebSocket Proxying - nginx in front of long-lived connections
• anyio and Trio — structured concurrency - AnyIO cancel scopes
• Pub/sub and Streams - Redis Streams as the event log
• LLM inference fundamentals - TTFT, and why streaming changes perceived latency
• Streaming - streaming agent runs