HF Streaming: SSE Mechanics and What to Watch For

OpenTechnologyApp's chatbot has two response modes: a normal request/response call, and a streaming variant that sends the reply token by token as it's generated. This guide is the streaming variant's actual wire format and the mistakes that break it.

The wire format

The streaming endpoint responds with Content-Type: text/event-stream. Each chunk of generated text arrives as its own event:

data: {"text": "some tokens"}

A blank line terminates each event — that's part of the SSE spec, not incidental formatting. When the response is complete, the server sends a sentinel event instead of closing the connection silently:

data: [DONE]

If something goes wrong mid-stream, the server sends an error event instead of just dropping the connection:

data: {"error": "some message"}

What this means for a client

A client reading this stream needs to:

1. Parse each data: ... line as its own unit, split on the blank-line terminator — not assume one chunk per network read. TCP and proxies can split or coalesce chunks in ways that don't line up with SSE event boundaries.
2. Check for the literal string [DONE] before trying to JSON.parse() the payload — it's not JSON, and parsing it as JSON will throw.
3. Handle the {"error": ...} shape as a terminal state, not as another content chunk to append to the visible reply.
4. Same gate as the non-streaming endpoint: if the org's chatbot is turned off (see the org-level on/off switch), this endpoint returns the same "not enabled" response before any streaming starts — a client should check for that up front rather than opening a stream connection speculatively.

Common client-side mistakes

• Buffering the whole response before rendering anything. Defeats the entire point of streaming — the reply should render incrementally as data: events arrive, not after the [DONE] sentinel.
• Not handling a dropped connection. SSE doesn't have automatic reconnect built into the browser EventSource API when used with a POST-based streaming endpoint like this one (native EventSource only supports GET). If you're building your own client rather than using the app's own UI, you need your own reconnect/retry logic — there isn't one for free.
• Assuming streaming skips the same failure modes as a normal request. It doesn't. A provider timeout, a refusal, or a routing decision all still apply — streaming only changes how the successful path delivers text, not whether the underlying request can fail the same ways described in the setup guide.

When to use streaming vs. the plain endpoint

Streaming is worth the extra client complexity when a human is watching the response render in real time — it makes a multi-second reply feel responsive instead of frozen. It's not worth it for a background job, an automation, or anything that consumes the full reply programmatically before doing anything with it — use the plain request/response endpoint there and skip the SSE parsing entirely.