Streaming an LLM answer into a React UI

Article.

How do you stream an LLM response from a FastAPI backend into a React UI so tokens render smoothly and Stop actually stops the model?

This is the frontend question a backend or AI engineer gets asked, because the interesting failures cross the boundary: a parser that works against one server and emits nothing against another, a component that re-renders a growing markdown string a hundred times a second, a Stop button that hides the text while the model keeps generating and billing, and a stream that ends halfway with no error.

The server side is in Server-Sent Events (SSE) (the protocol, buffering proxies, timeouts) and Streaming LLM responses over SSE (re-streaming the provider, Stop and billing). This note is the browser.

The wire format, briefly

Server-Sent Events is one long HTTP response of text/event-stream: fields on lines, events separated by a blank line.

event: token
id: 42
data: {"t": "Hello"}

: ping

event: done
data: {}

event: names the event, id: is the resume cursor, data: is the payload (several data: lines join with a newline), and a line starting with : is a comment, used as a keep-alive. Lines may end in LF, CRLF or CR.

EventSource, and why chat UIs do not use it

The browser’s EventSource parses all of that for you and reconnects, but it can only make a GET with no body and no custom headers. A chat turn is a POST with a JSON prompt and usually an Authorization header, so chat UIs use fetch and parse the stream themselves. What EventSource actually does, for the cases where it fits (notifications, job progress):

• Reconnects after a fixed delay, which the server sets with retry:; the spec does not require exponential backoff.
• Never reconnects after a non-200 status or a wrong Content-Type. A 401 from an expired session closes it for good (readyState becomes CLOSED), silently. A 204 is the intended way to tell it to stop.
• Sends Last-Event-ID on reconnect, so a server that keeps an event log can resume. With fetch you get none of this and rebuild it yourself.
• Cross-origin cookies need new EventSource(url, { withCredentials: true }).

Parsing a fetch stream correctly

The parser most tutorials show splits the text on "\n\n". Against a server that frames events with CRLF (sse-starlette does by default), it never finds a separator. A real run in Node, with the stream split into chunks at awkward points (one split lands inside the two bytes of “ü”):

naive, split on "\n\n": []
spec-shaped parser:
   token  "{\"t\":\"Grüße\"}"
   token  "line one\nline two"
   done   "{}"

The naive parser emits nothing until the stream closes, so the UI shows the whole answer at the end, exactly like a buffering proxy.

A parser that follows the spec

The essentials, in TypeScript:

const dec = new TextDecoder();                      // stream: true keeps split UTF-8
let buf = "", ev = { event: "message", data: [] as string[] };

function feed(chunk: Uint8Array, emit: (e: { event: string; data: string }) => void) {
  buf += dec.decode(chunk, { stream: true });
  const lines = buf.split(/\r\n|\r|\n/);           // all three line endings
  buf = lines.pop() ?? "";                          // keep the partial line
  for (const line of lines) {
    if (line === "") {                              // blank line: dispatch
      if (ev.data.length) emit({ event: ev.event, data: ev.data.join("\n") });
      ev = { event: "message", data: [] };
    } else if (!line.startsWith(":")) {             // ":" lines are comments
      const i = line.indexOf(":");
      const field = i < 0 ? line : line.slice(0, i);
      const value = i < 0 ? "" : line.slice(i + 1).replace(/^ /, "");
      if (field === "event") ev.event = value;
      else if (field === "data") ev.data.push(value);
    }
  }
}

In production use a maintained parser (eventsource-parser, or @microsoft/fetch-event-source, which also adds retries): the edge cases include a CR at the end of one chunk and an LF at the start of the next.

The hook: abortable, batched, honest about endings

The hook keeps three refs (unrendered tokens, the pending animation frame, the abort controller) and exposes start, stop, the text and a status.

export function useStreamedAnswer() {
  const [text, setText] = useState("");
  const [status, setStatus] = useState<Status>("idle");
  const pending = useRef("");                       // tokens not yet rendered
  const frame = useRef<number | null>(null);
  const ctrl = useRef<AbortController | null>(null);

  const flush = () => {
    frame.current = null;
    const chunk = pending.current; pending.current = "";
    if (chunk) setText((t) => t + chunk);
  };

Starting, stopping, ending (same hook)

  async function start(prompt: string) {
    ctrl.current?.abort();                          // one answer at a time
    const ac = (ctrl.current = new AbortController());
    setText(""); setStatus("streaming");
    let done = false;
    try {
      const res = await fetch("/api/chat", {
        method: "POST", signal: ac.signal,
        headers: { "content-type": "application/json" },
        body: JSON.stringify({ prompt }),
      });
      if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);
      for await (const ev of readEvents(res.body)) {   // the parser above
        if (ev.event === "token") {
          pending.current += JSON.parse(ev.data).t;
          frame.current ??= requestAnimationFrame(flush); // ≤ 1 render per frame
        } else if (ev.event === "error") {
          throw new Error(JSON.parse(ev.data).message);
        } else if (ev.event === "done") {
          done = true;
        }
      }
      setStatus(done ? "done" : "truncated");      // ended without "done"
    } catch {
      setStatus(ac.signal.aborted ? "stopped" : "error");
    } finally {
      if (frame.current !== null) cancelAnimationFrame(frame.current);
      flush();
    }
  }

  useEffect(() => () => ctrl.current?.abort(), []); // unmount stops the stream
  return { text, status, start, stop: () => ctrl.current?.abort() };
}

Four decisions in that code

• Stop is abort(), which closes the connection. Hiding the text without aborting leaves the model generating, and billing, to the end.
• Unmounting aborts too. Navigating away from the chat must not leave a stream running and calling setState on an unmounted component.
• Errors after the first byte arrive in-band. The status was already 200, so the server sends event: error; the client treats it as a failure.
• A stream that ends without done is truncated, not finished: a proxy timeout or a crashed worker. Show the partial answer with a retry, and report it, instead of presenting half an answer as complete.

What Stop does across the stack

1. abort() rejects the pending read with an AbortError and closes the connection (on HTTP/2, a reset of that one stream).
2. Each proxy closes its upstream; uvicorn reports http.disconnect.
3. Starlette cancels the response generator; CancelledError lands at its current await, inside the async with that holds the provider stream.
4. Leaving the async with closes the provider connection, and the provider stops generating. Tokens already in flight are still billed.

Step 4 only happens if the backend reads the provider inside the response generator. A provider read in a detached background task keeps generating after the user left; that bug, with a measured run, is in Streaming LLM responses over SSE.

Rendering cost

At 50-100 tokens per second, setText per token is 50-100 React commits a second, and a markdown renderer that re-parses the whole message on every commit does work proportional to the length each time: quadratic over the answer. Long answers make the tab sluggish, and input lag shows up in INP (Core Web Vitals).

• Batch per animation frame, as the hook does: at most about 60 renders a second however fast tokens arrive. useDeferredValue on the text is the React-native alternative.
• Split the message into blocks (paragraphs, code fences) and render completed blocks as memoised components; only the last, growing block re-parses.
• Keep scroll-follow cheap: scroll on frame, and stop following when the user scrolls up.

Measure with the React Profiler: commits per second and commit duration for a long answer, before and after.

Cancel or resume: choose one

A dropped connection and a deliberate Stop look identical to the server. Two designs:

• Coupled (the hook above): the run lives and dies with the request. Stop and tab-close end the bill; a network blip loses the answer.
• Decoupled: a worker generates into a log (a Redis Stream keyed by message id) and the client attaches with an offset, so a reload or blip resumes. Stop then has to be an explicit request (POST /runs/{id}/cancel), and abandoned runs need a timeout.

Say which one you built. The decoupled design is described in Streaming LLM responses over SSE.

Where it breaks in production

• Everything arrives in one burst: a buffering proxy, compression on text/event-stream, or a parser that does not recognise CRLF. Check the parser first: it fails identically in every environment.
• Streams cut at 30 or 60 seconds: idle timeouts (CloudFront 30 s, ALB and nginx 60 s) during a long tool call; the server must send keep-alive comments more often than the smallest one.
• A silent stop after a session expires: EventSource gets a 401 and closes forever; a fetch stream gets a 401 the code treats as an empty answer.
• “Stopped” but still billing: the UI hid the text without aborting, or the backend’s provider read was detached.
• A janky tab on long answers: per-token renders and full markdown re-parses.
• Six chat tabs, the seventh hangs: HTTP/1.1 allows six connections per host, and each open stream holds one; HTTP/2 multiplexes them.
• Half answers shown as complete: no done event checked.

The trade not taken

The Vercel AI SDK’s useChat (with its server helpers) instead of a hand-written hook. It gives a stream protocol, message state, tool-call rendering and abort out of the box, and is the right default in a Next.js app.

The cost is coupling the client and server to its protocol and versions, and less control over batching, resume and error semantics, which are the parts this note is about. WebSocket is the choice when the client talks during the stream (voice, interrupting with new input); for a one-way answer it adds reconnect, auth and proxy work for nothing.

Related

• Server-Sent Events (SSE) - the protocol and the server side
• Streaming LLM responses over SSE - provider re-streaming, Stop and billing, resume
• Race conditions and AbortController - AbortController and stale responses
• WebSockets — Integration, Reconnection, and Real-Time UX - when the client needs to talk back
• Design: Chat (Optimistic UI + WebSocket + Presence) - designing a chat frontend
• Core Web Vitals - INP and render cost