A Latency Scorecard for Streamed Answers: First Chunk and Tail

Ambiguity kills these numbers faster than bad infrastructure. Pin down each definition in one sentence and put it in the dashboard subtitle.

Time to first chunk (TFC). Timestamp of the first chunk at the client minus timestamp of the request at the client. Not at the server. Server-side TFC excludes network transit and DNS, which is fine for capacity planning and wrong for user experience. Record both if you can, and label them.

The gap. For each chunk after the first, subtract the previous chunk's arrival timestamp. Then report p50 and p95 across all intervals in the window. Do not include the interval between the request and the first chunk, and do not include the interval between the last chunk and the end event. Those are TFC and a separate closing delay.

The tail. Timestamp of the last chunk (or the explicit end event) minus timestamp of the first chunk. Note what this excludes: the pre-first-chunk wait. That is deliberate. Tail answers "once it started, how long did it run?"

A fourth number is useful but optional: the closing delay, from last content chunk to the end or done marker. If a provider sends a terminal event with no content, this number can be large and it inflates naive end-to-end measurements. AIPerf explicitly strips the final done signal so it does not enter end-to-end latency per NVIDIA, and you should do the same.

On units: timestamp every event with a monotonic clock in milliseconds, client-side. performance.now() is monotonic and unaffected by wall-clock adjustments, which matters when a tab sleeps and wakes. If you log in the browser, measure in the browser.

Assume a log with these fields per request: request_start, an array of chunk send times, and end_time. Take one request. Chunks arrived at 412, 452, 497, 541, 588, 634, 691, 950, 998, 1044 ms after the request started, and the end event landed at 1051 ms.

First, TFC. The request started at 0, the first chunk at 412 ms. TFC = 412 ms.

Next, intervals. Five of the nine are 40 to 47 ms. One is 57 (634 to 691). One is 259 (691 to 950). Sorted: 40, 44, 45, 47, 46, 47, 57, 259, 46. That gives p50 = 46 ms and p95 = 259 ms. Gap = 46 ms / 259 ms.

Then the tail: 1044 − 412 = 632 ms. Tail = 632 ms. Closing delay is 7 ms and does not enter any of the three metrics.

Now the interpretation. The p50 gap is healthy. The p95 gap is 259 ms, and it comes from a single stall between chunk 7 and chunk 8. One stall in a ten-chunk stream is not noise you can average away, and it is exactly the kind of thing a mean interval erases. That single stall is what a user perceives as the response freezing.

Here is the reduction in a form you can drop into a script:

def scorecard(t_start, chunk_times, t_end):
    first = chunk_times[0] - t_start
    gaps = sorted(b - a for a, b in zip(chunk_times, chunk_times[1:]))
    def pct(vals, p):
        idx = min(len(vals) - 1, int(round((p / 100) * (len(vals) - 1))))
        return vals[idx]
    return {
        "tfc_ms": first,
        "gap_p50_ms": pct(gaps, 50),
        "gap_p95_ms": pct(gaps, 95),
        "tail_ms": chunk_times[-1] - chunk_times[0],
        "close_ms": t_end - chunk_times[-1],
    }

Run it over a day of logs and aggregate the per-request results into p50 and p95 across requests, never by averaging the per-request values. Averaging percentiles is a common and quiet error.

One budget for every query is the fastest way to make a scorecard useless. Split traffic into classes and set each budget from what the class is for.

The "long generation" row encodes a real constraint: once streaming outpaces a reader, further gains in the gap are invisible. Silent reading of non-fiction runs around 238 words per minute, roughly 5 to 6 tokens per second, per the meta-analysis cited in ClickHouse's guide. A stream at 20 to 40 tokens per second is comfortably beyond reading speed, so chasing a 20 ms gap on a long answer buys nothing. Spend that effort on TFC instead, because that is the silence the user actually sits through.

NVIDIA's guide notes that TTFT includes request queueing, prefill and network latency, and grows with prompt length in the NIM metrics reference. That is the mechanism behind the TFC budget: if your short-lookup class carries a large system prompt or a long retrieved context, TFC will exceed 600 ms regardless of model choice. For the short-lookup class, treat a TFC near one second as a routing signal: the query was misclassified and should have gone out as a long generation.

A resume is not a retry. The client reconnects, asks for the stream from a known point, and the server replays what was missed. The machinery is well defined for Server-Sent Events: the browser stores the id field of the last dispatched event and sends it back in the Last-Event-ID request header on reconnect per the HTTP header reference. The server reads the header and resumes from the correct position.

That single header reshapes all three numbers.

First chunk. After a resume, TFC restarts at the new request. If your dashboard labels this as TFC, your p95 TFC will look catastrophic on flaky networks and you will chase a problem that is not in your model. Fix: report resumed requests under their own class, and either measure the reconnect handshake separately or inherit the original TFC. The user's experience of "when did text appear" belongs to the first connection.

The gap. The reconnect interval is a gap of seconds, and it is not an inter-chunk interval in any meaningful sense. Count it separately as a resume stall. If you do not, one dropped connection will dominate your p95 gap for the whole window. During the reconnect, events can be produced that the client has not yet seen, which is exactly the loss case teams hit when SSE connections are recycled on an idle timeout as described here.

The tail. Two outcomes. If the server replays the missed chunks and the client re-renders them, the tail runs from the original first chunk to the final chunk, and it grows by roughly the reconnect duration. If the server resumes at the boundary and the client keeps its buffered text, the tail is close to the original. Either way, report the tail with a replay flag, because a tail that grew by 3 seconds due to a WiFi handoff tells you about mobile networks, not about your model.

The practical checklist for resume instrumentation:

1. Tag every request with resumed: true/false and the Last-Event-ID sent.
2. Log the reconnect timestamp and the first chunk time after reconnect as a separate resume_handshake_ms.
3. Log how many chunks were replayed and whether any were duplicates.
4. Exclude the reconnect interval from the gap distribution.
5. Report the tail with and without the reconnect duration.

Point 4 matters more than it sounds. This testing guide on SSE reconnection lists the concrete failure modes to test for: ids attached to incomplete events, off-by-one replay queries, stale retention windows, and proxy buffering that makes a healthy stream look silent. Each of those produces a number that looks like a latency regression and is actually a correctness bug. An event buffer that has evicted the requested id forces a full replay or a snapshot refresh, and that shows up in the tail as a large jump; a request for an id older than anything retained has to have a defined behavior, either an error event or a reset marker as this SSE resumption design notes.

If your client cancels the in-flight read on resume, remember that cancelling a locked stream throws a TypeError per MDN's cancel documentation, and that breaking out of for await cancels the underlying stream unless you pass preventCancel per the WHATWG streams discussion. Getting that wrong produces a "resume" that silently starts from zero, which is a tail measurement that means nothing.

Bring three columns to the review, one per query class, and resist the urge to add a fourth. Each cell should show a p50 and a p95 against its budget. Anything at or under budget gets no discussion. The conversation goes to the cells that are over.

Use this order when a cell is red:

1. Check the tail against output length first. If median output length in that class grew, the tail moved for a boring reason. Nothing is broken.
2. Check the gap p95 against gap p50. A wide spread (say 45 ms p50, 400 ms p95) is stalls. A narrow but high pair (200 ms p50, 240 ms p95) is a slow model or heavy batching.
3. Check TFC against prompt size. TFC that tracks prompt length is a prefill problem. TFC that is flat and high across prompt sizes is queueing or buffering.
4. Check the resume rate last. If more than a small fraction of requests in a class are resumes, fix reconnection before you tune anything, because you are measuring the network.

Two measurement hygiene rules complete the picture. Never average percentiles across requests, and never measure TFC server-side for a user-facing number. If you want one extra panel, add "share of requests that stalled over 250 ms at any point," which catches the single-stall case the worked example showed and which percentiles can smooth over when the stall count is low.

A final word on where the numbers come from. If you parse incrementally on the client, the render cost lands inside your gap measurement. A stateful incremental parser handles a 12 KB document in tens of milliseconds where re-parsing the whole prefix on every chunk degrades quadratically and becomes unusable around the 5 KB mark per this measured comparison. Libraries differ here: some mutate the parsed object in place and document the invariants behind that choice per JSON River's discussion, while others return a best-effort partial object on every chunk per best-effort-json-parser or expose a typed stream iterator per jsonchunk. Pick one deliberately, then measure: the parser is part of your gap budget whether you planned for it or not.

If your streamed answers now span several model providers, the metric definitions have to stay identical across them or the scorecard stops being comparable. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API with prepaid USD pricing, which keeps instrumentation identical when you move a query class to a different model: https://quantaintelligence.ai/inference