Reconnecting a Stream Without Duplicating Tokens

Every chunk the server emits gets a sequence number assigned at one place in the pipeline, immediately before it enters the per-stream buffer. Three rules make the numbering usable:

1. One writer per stream. If two goroutines, tasks or threads can append to the same stream, a lock or a single-writer queue must serialize the assignment. Concurrent publishers that each grab "the next number" produce commit-order disagreements, which is precisely the invariant an SSE test should check: each accepted ID is greater than the previous one (SSE reconnect testing).
2. Monotonic, not necessarily contiguous. Sparse numbering (gaps from filtered chunks, for example) is fine as long as the server documents that replay is "strictly greater than the supplied number." Contiguous numbering is easier to test and easier to reason about in logs, so prefer it unless you have a reason not to.
3. The number travels twice. Put it in the SSE id: field and in the JSON body of the chunk. The Last-Event-ID reference notes the header value is a UTF-8 string matching the id field, and a widely cited Stack Overflow answer on lost SSE events recommends including the same identifier inside the data packet as well, so that non-SSE transports and client-initiated reconnects can carry it (SO answer on Last-Event-ID).

A chunk on the wire looks like this:

id: 151
event: token
data: {"seq":151,"delta":" production","done":false}

The blank line is part of the protocol, not decoration: SSE dispatches an event only when a blank line terminates it, and a client that records the id the moment the line arrives rather than at dispatch will reconnect from a position it never actually processed. That off-by-one is the single most common duplicate-token bug in hand-rolled clients, and it is worth an explicit test at the parser level (SSE reconnect testing).

The resume token is the client's claim about what it has. Keep it dumb and verifiable: an opaque string that the server can decode into (stream_id, last_seq). Do not put secrets in it and do not sign it if you can avoid it; a UUID stream identifier plus an integer sequence is guessable only by someone who already knows the stream ID, and stream IDs should not be authorization tokens anyway.

Two delivery paths need to work:

• Automatic, for browsers. When an EventSource connection drops, the browser reconnects and sends the stored last event ID in the Last-Event-ID request header, so the server can resume without any client code (Last-Event-ID). This is the cheapest possible client contract: it works even if your client code has bugs, because the browser does the remembering.
• Explicit, for everything else. Mobile clients, fetch-based readers and SDKs pass ?cursor=<token> on the reconnect request. Ably points out that SSE's built-in header is only the handshake piece of a four-piece problem, and that on distributed infrastructure a reconnecting client can land on an instance that has never heard of the session. The cursor parameter is also what lets a client persist its position across an app restart.

If both are present, the explicit cursor wins and should be validated against the header. Log a mismatch rather than erroring on it, because a mismatch usually means the client's stored position is fresher than the browser's.

That last row is the one people forget. A cursor from stream A presented on stream B must fail loudly. Silently starting fresh looks like a resume to the client and produces a duplicated response.

The buffer is a fixed-capacity circular structure per stream, holding the unacknowledged tail. You do not need per-token acknowledgments: humans read slower than models generate, so treat "delivered to a live connection" as acknowledged and let the buffer cover only the gap between server writes and client receipt.

Sizing is arithmetic, not vibes. Take the worst realistic case:

• Longest response you intend to survive: 4,000 tokens.
• Bytes per chunk, including SSE framing and JSON wrapper: about 45 bytes for a small delta, call it 60 with overhead.
• That is roughly 240 KB of buffered payload for a full response retained from the start.

Now the decision: do you keep the whole response, or only the last N chunks? Keep the whole thing while the stream is open, because the interesting failure (tab suspended for two minutes, connection reaped by a proxy) can happen at any point, and a buffer that only holds 500 chunks may have evicted the position the client needs. The Sentry Spotlight issue that specifies this feature for their sidecar settles on a 500-event circular buffer and then explicitly lists buffer overflow as an edge case to handle, with the client expected to do a full state reset (Sentry Spotlight #1095). That is the right shape: bounded buffer, documented bound, loud failure past it.

Retention after the stream ends is separate. Keep the buffer for a fixed grace period after the final chunk (a few minutes is generous for a mid-response drop and cheap in memory), then evict. Track two counters per stream: highest_seq and oldest_retained_seq. Any resume request below oldest_retained_seq is a 409.

Here is the shape that keeps a reconnecting client correct.

First request.

POST /v1/stream HTTP/1.1
Content-Type: application/json

{"prompt": "...", "stream": true}

Respond with Content-Type: text/event-stream, and emit a first control event carrying the stream ID and the cursor seed:

id: 1
event: start
data: {"stream_id":"9f2c...","seq":1}

Reconnect, browser style. The EventSource cannot replay a POST, so a browser-reconnectable stream needs a GET endpoint keyed by stream ID:

GET /v1/stream/9f2c... HTTP/1.1
Last-Event-ID: 151

Reconnect, SDK style.

GET /v1/stream/9f2c...?cursor=eyJzIjoiOWYyYyIsImwiOjE1MX0 HTTP/1.1

Server-side handling, in order. Read the cursor (explicit parameter, else Last-Event-ID). Decode it. If it does not decode, return 400. If it names a different stream, return 403. Look up the buffer. If the requested sequence is older than oldest_retained_seq, return 409 with a body naming the oldest available sequence, so the client can decide between a full restart and fetching a snapshot. Otherwise write every retained chunk with seq > last_seq in order, then attach the connection to the live feed, with an overlap check so a chunk emitted during replay is not sent twice.

The 409 body.

{"error":"cursor_expired","oldest_available":812,"action":"restart"}

Clients that receive this should discard the partial and start a fresh stream, which the companion article handles. The important property is that the failure is explicit: a 409 is far better than a 200 that quietly starts at a later point, because a silent skip is indistinguishable from a successful resume on the client side.

Heartbeats. Send SSE comment lines (: ping) on an interval shorter than your proxy's idle timeout. Comments do not dispatch events and must never advance the client's stored ID, so they cannot create phantom resume positions (SSE reconnect testing). The classic production incident here is a load balancer closing idle connections at a fixed interval, which is exactly the situation described in the Stack Overflow question about lost events during five-minute connection recycling (SO #58335110). Heartbeats plus a buffer turn that incident into a routine reconnect.

The logic fits in a few dozen lines in any language. This is TypeScript-style pseudocode for the reconnect handler, written to show the order of checks rather than a framework integration.

function resume(streamId: string, cursor: string, headers: Headers, res: Response) {
  const decoded = decodeCursor(cursor || headers.get('last-event-id') || '');
  if (!decoded) return send(res, 400, { error: 'bad_cursor' });
  if (decoded.streamId !== streamId) return send(res, 403, { error: 'wrong_stream' });

  const stream = registry.get(streamId);
  if (!stream) return send(res, 404, { error: 'unknown_stream' });
  if (decoded.seq < stream.oldestRetainedSeq) {
    return send(res, 409, {
      error: 'cursor_expired',
      oldest_available: stream.oldestRetainedSeq,
      action: 'restart'
    });
  }

  openEventStream(res);
  const replayed = new Set<number>();
  for (const chunk of stream.buffer.after(decoded.seq)) {
    writeEvent(res, chunk.seq, chunk.event, chunk.data);
    replayed.add(chunk.seq);
  }
  stream.subscribe((chunk) => {
    if (replayed.has(chunk.seq)) return;   // overlap guard
    writeEvent(res, chunk.seq, chunk.event, chunk.data);
  });
}

Two details carry most of the correctness. The overlap guard prevents the double-send that happens when a live chunk lands between buffer read and subscription, and the strict after(decoded.seq) (strictly greater, never greater-or-equal) prevents the boundary duplicate. An SSE test invariant worth writing down: the first resumed ID must be greater than the supplied Last-Event-ID (SSE reconnect testing).

You could accept a bare integer as the cursor. The reasons to wrap it in a token are practical:

• The token can carry the stream ID, so a cursor from another stream fails validation instead of landing on the wrong buffer.
• The token can carry a format version, so you can change the encoding later without breaking deployed clients.
• The token is opaque, so clients stop treating it as a number they can increment, which they will do, with predictable consequences.

Encode a small JSON object as base64: {"v":1,"s":"<stream_id>","l":151}. Keep the field names short because the token travels in a query string on every reconnect. Validate the version, then the stream ID, then the sequence. The sequence field should be the last sequence the client dispatched to its parser and rendered, not the last one it received, which is the same distinction that makes the SSE id field safe.

One more decision: is the cursor per stream or per conversation? Per stream is simpler and matches the lifetime of the ring buffer. If a conversation spans multiple streamed responses, the client stores a cursor per response and passes it only when resuming that response.

Three levels of test cover this, and all three catch different defects: a parser-level test for framing, an in-process integration test for replay state, and a test through the same reverse proxy used in production (SSE reconnect testing). The integration test is the one that catches the bugs a code review misses. Drive it from a script that opens a stream, records the first N chunks, kills the connection, reconnects with the recorded cursor, and asserts that the concatenation of both halves equals the uninterrupted response byte for byte. Then loop N from 0 to the final sequence; the run where N equals the final sequence catches the "reconnect after completion" case, and the run where N is one below oldestRetainedSeq catches the 409 path.

Checklist for the server side:

Error handling on the client side follows from this contract. Treat 409 as "restart," 400 as a client bug worth logging, and 5xx or a timeout as "retry with the same cursor after backoff." Retrying with an unchanged cursor is safe precisely because replay is idempotent by sequence: the point of the whole design is that the client never has to guess what the server already sent (error handling and retries for LLM calls).

If you are building the model service behind this contract rather than the transport, the other half of the problem is that the token generator itself must support these semantics. Quanta Inference (https://quantaintelligence.ai/inference) exposes text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid US dollar pricing, so you can put your own sequenced, buffered stream layer in front of any of them without rewiring providers. The resume contract above is yours to own either way; the buffer, the cursor and the 409 are application logic, not model infrastructure.