Latency, Throughput & SLOs — Inference Engineering

01

Two clocks: TTFT and TPOT

TL;DR · Users feel two latencies: how long until the first token appears (TTFT) and how fast tokens stream after that (TPOT). Prefill sets the first; decode sets the second.

A single number can't describe how a stream feels. A reply that starts instantly but dribbles out, and one that pauses then floods — those are different experiences, and you need two clocks to tell them apart.

TTFT — the wait before anything appears

TTFT (time to first token) is the latency a user feels waiting for the first response token from a prompt. It's set by the prefill phase — the one big pass over the whole prompt — so longer prompts push TTFT up.

TPOT — the cadence of the stream

TPOT (time per output token), also called inter-token latency (ITL), is the steady interval between successive output tokens. It's set by the decode phase — the one-token-at-a-time loop — so it governs how smoothly the answer streams.

Keywords — tap to unfold the plain meaning

Analogy ❝Think of a line cook fetching from a faraway pantry. TTFT is the first long walk to the back room to gather everything the order needs — one big trip, set by how much the ticket asks for. TPOT is the rhythm of plating each bite afterward: a steady there-and-back for every token, bounded by how fast the cook can shuttle to memory and return. Prefill is the gather; decode is the shuttle.

TTFT (set by prefill) is the wait for the first token; TPOT / ITL (set by decode) is the gap between tokens after that. Two clocks, two phases.

02

Throughput — and why you read the percentiles

TL;DR · Throughput is a system-wide rate (tokens/sec or requests/sec). Latency is per-request — so you track it at percentiles like p50, p95, p99, not as an average.

Latency and throughput aren't two views of one thing; they pull against each other. Pack the box for throughput and tail latency creeps up — so you watch the tail, not the mean.

Throughput is the fleet's rate

Throughput is the system-wide rate of work, measured in tokens/sec or requests/sec (QPS). It's what a replica delivers in aggregate across all the requests it's serving at once — a property of the box under load, not of any one request.

Latency lives in the tail

A single slow request hides inside a healthy average. So latency SLOs are stated as percentiles — p50 (median), p95, p99. "p99 TTFT under 500 ms" means 99 of every 100 first-token waits clear that bar. The tail is what users actually complain about.

Keywords — tap to unfold the plain meaning

Throughput = system rate (tokens/sec, requests/sec). Latency = per-request, read at p50 / p95 / p99. Watch the tail, never the average.

03

The knee — where latency turns into queueing

TL;DR · The knee is the load level where throughput saturates but latency starts climbing steeply. Run just below it; past it you mostly buy queueing delay, not more work done.

There's a single point on the load curve that decides everything about how you run a replica. Find it, stay under it, and the system feels fast. Cross it and the same hardware suddenly feels broken.

Below the knee: load buys throughput

Add load and the replica does more useful work — throughput rises while latency stays flat. This is the region you want to live in: spare capacity converts directly into served requests.

Past the knee: load buys only delay

At the knee, throughput plateaus. Push harder and latency explodes while the work done barely moves — the extra requests just sit in a queue. You "mostly buy queueing delay, not more work done."

Below the knee, more load means more throughput at flat latency. At the knee throughput saturates; past it latency climbs steeply while throughput stays put.

Keywords — tap to unfold the plain meaning

Analogy ❝Picture one road into a kitchen's faraway pantry. While it's under capacity, every extra cook you send fetches more food — output rises, the trip still takes the same time. But there's a knee: once the road is packed, a new cook doesn't fetch more, they just wait in line behind the others. The pantry isn't producing more; the cooks are only queueing. You want to load the road right up to that point and not one cook past it.

The knee is where throughput saturates and latency turns steep. Run just below it — past it, extra load buys queueing delay, not work.

04

Goodput — throughput that actually meets the SLO

TL;DR · Goodput is the throughput that meets both the TTFT and TPOT SLOs. A request served too late is a failure, not a success — so raw throughput can stay flat while goodput collapses.

Past the knee, the dashboard can lie to you. Tokens/sec looks fine, the box looks busy — but every one of those tokens is arriving late. The honest metric counts only the work that met its deadline.

An SLO is a promise with a number

An SLO (service-level objective) is a target you commit to — e.g. "p99 TTFT < 500 ms" and "p95 TPOT < 40 ms". Each request either meets both thresholds or it doesn't. There's no partial credit for a token that arrived late.

Goodput counts only the wins

Goodput is "the throughput that meets both the TTFT and TPOT SLOs." Requests served late are failures, not successes. Push past the knee and raw throughput may look stable while goodput quietly collapses — every served request is now an SLO violation.

Keywords — tap to unfold the plain meaning

SRE note Optimize goodput, not throughput. A replica past its knee can report healthy tokens/sec while serving every request late — raw throughput stable, goodput near zero. This is the result behind DistServe (Zhong et al., 2024) and Sarathi-Serve (Agrawal et al., OSDI 2024): tune the system to maximize requests that meet both SLOs, and let raw throughput be whatever it is.

Goodput = throughput that meets both the TTFT and TPOT SLOs. Late requests are failures. Past the knee, throughput can hold while goodput collapses.

05

Little's Law sizes the fleet

TL;DR · Little's Law ties in-flight requests, throughput, and latency together — so a concurrency cap and a measured latency tell you the per-replica QPS and how many replicas you need.

Capacity planning isn't guesswork. One small law links the three quantities you already measure, and it turns "how many GPUs?" into arithmetic you can do on a napkin.

In-flight = throughput × latency

Little's Law says the number of requests in flight equals throughput times latency. Rearranged, a replica's concurrency cap divided by its latency gives the maximum throughput it can sustain.

From per-replica QPS to replica count

The vLLM flag --max-num-seqs is that concurrency cap. Divide it by average latency (measured at the knee) to get per-replica QPS, then divide peak demand by that to get how many replicas to run.

Math, decoded

L = X × W → X_max = concurrency caplatency → replicas = peak QPSper-replica QPS

Lin-flight requests — how many are being served at once (Little's Law)
Xthroughput — requests completed per second (QPS)
Wlatency — average time one request spends in the system
concurrency capthe replica's --max-num-seqs — max sequences it runs in parallel
peak QPSthe busiest demand you must serve; divide by per-replica QPS for replica count

In-flight requests = throughput × latency. So max throughput = concurrency cap ÷ latency, giving per-replica QPS at the knee. Then replicas needed = peak QPS ÷ per-replica QPS.

Keywords — tap to unfold the plain meaning

Analogy ❝If the pantry road holds 64 cooks at once (the concurrency cap) and each round trip takes a fixed time, the road's output is fully determined: 64 divided by that trip time is the most orders it can finish per second. Want to serve twice the dinner rush? You don't speed up the road — you build a second one. Replica count is just peak orders divided by what one road delivers.

Little's Law: in-flight = throughput × latency. Max throughput = concurrency cap ÷ latency. Replicas = peak QPS ÷ per-replica QPS at the knee.

06

On your cluster

TL;DR · vLLM exports the exact metrics for this lesson — TTFT, TPOT, running and waiting requests — and flags that protect latency under load. Autoscale on the queue, size with Little's Law.

Every idea here maps to a number vLLM already exports on the 4×H100 box. You don't have to infer the knee — you can watch the queue grow and the tail rise in real time.

The four metrics that matter

Scrape time_to_first_token for the TTFT SLO and time_per_output_token for the TPOT SLO. Watch num_requests_running (live concurrency) and num_requests_waiting — a growing queue is the clearest "you're past the knee, autoscale now" signal.

Flags that protect the tail

--max-num-seqs is your concurrency cap for Little's Law. Turn on --enable-chunked-prefill with a --max-num-batched-tokens budget so big prefills don't stall the decode loop — this is what keeps TTFT and TPOT honest under load.

Watch the knee on the 4×H100 box — vLLM /metrics

curl -s localhost:8000/metrics | grep -E 'num_requests_(running|waiting)|time_to_first_token|time_per_output_token'

# launch flags that cap concurrency and protect latency:
vllm serve Qwen3.6-27B-FP8 \
  --max-num-seqs 64 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192

Cluster note On the 4×H100 box, drive load up while watching num_requests_waiting: while it stays near 0 you're below the knee and throughput keeps rising; the moment it climbs and time_to_first_token P99 elevates, you're past the knee — autoscale on num_requests_waiting (or TTFT-P99 growth), route to keep each replica below its knee, and size the fleet with Little's Law (--max-num-seqs ÷ latency).

Keywords — tap to unfold the plain meaning

Check yourself

Which phase sets TTFT, and which sets TPOT?
What happens to throughput, latency, and goodput as you push past the knee?
You measure 64 concurrent slots and an average latency — how do you turn that into per-replica QPS and a replica count?
Which vLLM metric is the clearest "autoscale now" signal, and why?

Scrape TTFT, TPOT, running and waiting from vLLM; cap with --max-num-seqs and chunked prefill; autoscale on num_requests_waiting; size with Little's Law.