Benchmarking Correctly — Inference Engineering

01

A number without a workload is not a benchmark

TL;DR · A bare tokens/second figure means nothing on its own. The point of this lesson is to build benchmarks that are reproducible and to reject comparisons that aren't real.

"Our server does 10,000 tokens/s" sounds impressive until you ask: which model, which prompt lengths, how many requests at once, and did it meet any latency target? Strip those away and the number is just a vibe.

The one rule everything hangs on

A benchmark is a measurement you can defend and repeat. The core principle of this whole lesson: a number without a workload is not a benchmark. A throughput figure with no stated workload, latency target, or quality bar is unfalsifiable — nobody can reproduce it or argue with it.

Peak throughput is the classic trap

Peak tokens/s measured with no latency constraint and no quality constraint is, in the lesson's words, not a serving result. A server can hit huge throughput by piling requests into giant batches — while every individual user waits far too long for their first word.

Five steps make it real

The rest of this lesson is five steps that turn a raw number into a defensible benchmark: write the workload contract, warm up deliberately, choose your load model, report distributions and goodput, and change only one variable at a time.

Keywords — tap to unfold the plain meaning

Analogy ❝"This kitchen plates 600 dishes an hour" tells you nothing until you ask: 600 of what — toast, or beef Wellington? With how many cooks, and did the food arrive hot or stone cold? A line cook can plate furiously fast by letting every ticket sit in a giant pile and serving them all at the end — great throughput, miserable diners. The headline number only means something once the order is written down.

A number without a workload is not a benchmark. Peak tokens/s with no latency and quality constraints is not a serving result — it's a vibe.

02

Write the workload contract first

TL;DR · Before you run anything, write down exactly what you're testing: model, hardware, precision, prompt and output lengths, sampling, cache state, load, and streaming. That document is the contract.

The contract is the difference between a benchmark someone can reproduce and a screenshot they have to trust. If it isn't written down before the run, it didn't happen.

What the contract must record

The workload contract documents: model and engine revisions, hardware, precision, prompt/output length distributions, sampling settings, cache state, concurrency or arrival rate, and streaming mode. Anything that changes the result and isn't written down is a hole someone will fall into.

Input and output tokens stress different phases

Prompt length and output length aren't interchangeable: input tokens stress prefill, output tokens stress decode. A long-prompt / short-answer job and a short-prompt / long-answer job hammer completely different parts of the engine.

Same model, two different worlds

A 128/128 run (128 input, 128 output tokens) is a fundamentally different test from a 4K/32 run (4,000 input, 32 output). Comparing two servers across mismatched length profiles isn't a comparison at all.

Keywords — tap to unfold the plain meaning

Length profile decides which phase you're really measuring. 128/128 spreads work across prefill and decode; 4K/32 is almost all prefill. Both are valid — but they answer different questions.

SRE note Pin every revision. "Model X on engine Y" isn't enough — record the exact model and engine commit/version, the precision (e.g. FP8), and cache state (cold vs warm prefix cache). A benchmark you can't reproduce from the written contract is a story, not data.

Write the contract first: model/engine revisions, hardware, precision, length distributions, sampling, cache state, load, streaming. Input stresses prefill; output stresses decode.

03

Warm up deliberately

TL;DR · The first requests are slow for reasons that have nothing to do with steady serving. Measure cold-start and steady-state separately — never silently mix or discard them.

If you average the very first request in with the rest, you're blaming the engine for a one-time setup cost. If you throw away cold-start entirely, you're hiding a number some users actually feel.

Why the first runs are slow

First runs include model loading, compilation, allocator growth, and cold caches. None of that recurs once the server is warm — so it tells you about startup, not about steady serving.

Two questions, two tests

Separate startup behavior from steady-state serving. They answer different questions: "how long until this replica is ready?" versus "how does it perform once it's running?"

Don't silently discard either

The trap is dropping cold-start data without saying so. If users hit a freshly-scaled replica, that cold latency is real to them. Report both phases honestly rather than quietly deleting whichever is inconvenient.

Keywords — tap to unfold the plain meaning

Analogy ❝The first ticket of the morning is slow because the cook is still firing up the burners, pulling pans, and the walk-in is freezing cold — the food has to come from a faraway pantry before anything's prepped at the line. That first slow plate isn't how lunch service runs. But if a customer walks in at 6 a.m. sharp, they live that cold-start. So you time the warm-up and steady service separately — and you don't pretend the cold first hour never happened.

First runs include model loading, compilation, allocator growth, and cold caches. Measure startup and steady-state apart — and never silently discard the one that matters to users.

04

Closed-loop or open-loop load

TL;DR · A closed-loop client waits for each reply before sending the next; an open-loop client fires requests at a fixed external rate. They answer fundamentally different questions — pick on purpose.

Choose the wrong load model and your benchmark can look healthy while the server is actually drowning. The difference comes down to who's allowed to wait: the client, or the queue.

Closed-loop hides overload

A closed-loop client sends a new request only after the previous one finishes. Because it slows down exactly when the server does, it can hide overload through coordinated omission — the requests that would have piled up never get sent, so the tail latency never shows.

Open-loop exposes the queue

An open-loop client's arrivals follow an external arrival rate, independent of how the server is doing. When the server falls behind, requests pile into the queue — so open-loop exposes queues and tail latency that closed-loop quietly absorbs.

Two different questions

Closed-loop answers "how fast can a fixed set of clients churn through work back-to-back?" Open-loop answers "what happens to latency when traffic arrives at rate R regardless of my state?" Production traffic is open-loop, so SLO testing usually should be too.

Keywords — tap to unfold the plain meaning

Closed-loop clients back off exactly when the server struggles, so the tail never appears. Open-loop clients keep arriving at rate R, so a slow server builds a visible queue — the latency real users would feel.

Closed-loop waits for each reply and can hide overload via coordinated omission. Open-loop arrives at a fixed rate and exposes queues and tail latency. Pick the one that matches your question.

05

Report distributions and goodput

TL;DR · Report full distributions — TTFT, TPOT/ITL, end-to-end latency, throughput, queue time, errors — at p50/p95/p99. Then headline goodput: the throughput that actually met the SLO.

A single average hides the users who suffer. The whole reason to report percentiles is that the worst 5% of requests are where your reputation lives — and goodput is the number that refuses to count work that missed its target.

The metrics that matter

Report TTFT (time to first token), TPOT / ITL (time per output token, a.k.a. inter-token latency), end-to-end latency, input/output throughput, queue time, and errors. One mean per metric is not enough.

Always show percentiles

Give each metric as p50 / p95 / p99 — the median, the 95th, and the 99th percentile. The p99 is the experience of your unluckiest users, and it's usually where SLOs are written and broken.

Goodput is the honest headline

Goodput counts only requests meeting the SLO. A run can post huge raw throughput while half its requests blew past their latency target — those don't count. Goodput is throughput with the quality-of-service bar enforced.

Keywords — tap to unfold the plain meaning

Math, decoded

goodput = requests meeting the SLOtotal requests × throughput

requests meeting the SLOhow many finished requests stayed inside every latency target (e.g. TTFT and TPOT bars)
total requestsevery request that was sent during the measured window, met or not
throughputraw requests- or tokens-per-second the server produced
goodputthe useful rate — work that actually satisfied the SLO; missed requests are not counted

Goodput keeps only the slice of throughput that met the SLO. If raw throughput is high but goodput is low, the server is fast at producing answers nobody got in time.

Math, decoded — TPOT from end to end

TPOT = end-to-end latency − TTFTN − 1

end-to-end latencytotal wall-clock time from sending the request to receiving the final token
TTFTtime to first token — how long until the very first output token arrives
Nnumber of output tokens generated for this request
N − 1the gaps between tokens after the first one — that's what TPOT averages over

After the first token lands (TTFT), the remaining time is spread across the other N−1 tokens. TPOT (equivalently ITL) is that per-token gap — the steady-state typing speed users feel.

4×H100 cluster note The lab's stored values — 3,288 prompt tok/s and ~108 generation tok/s from the live cluster — are only directional evidence. The exact model/engine revisions and prompt/output token lengths weren't retained, so there's no full workload contract behind them. Real, comparable numbers; not a reproducible benchmark — exactly why Station 02's contract matters.

Report TTFT, TPOT/ITL, end-to-end latency, throughput, queue time, errors — all at p50/p95/p99. Then lead with goodput: only the requests that met the SLO count.

06

Change one variable at a time

TL;DR · Hold the workload and the output quality constant, change exactly one thing, repeat the runs, report the variance, and keep the commands plus raw results so anyone can rerun it.

If you change the model, the batch size, and the prompt mix all at once and the number moves, you've learned nothing about why. A clean benchmark moves one lever per experiment.

Hold everything else still

Hold workload and quality constant, then change a single variable. If you don't pin output quality too, you might be "winning" only because the new setting quietly degraded the answers.

Repeat and report variance

Run it more than once and report variance. A single run can't tell signal from noise; showing the spread across repeats is what makes a difference believable.

Keep commands and raw results

Retain the commands plus the raw results. A benchmark nobody can rerun from your saved invocation and outputs is a claim, not evidence — and "trust me" is not reproducible.

A reproducible run keeps the invocation, not just the headline

vllm bench serve \
  --model Qwen3.6-27B-FP8 \
  --dataset-name random \
  --random-input-len 128 --random-output-len 128 \
  --request-rate 8 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,95,99 \
  --save-result --result-filename run.json

Why this matters The flags are the workload contract from Station 02 made executable: model + precision, the 128/128 length profile, an open-loop --request-rate, and ttft,tpot,itl,e2el reported at 50,95,99. --save-result retains the raw output so the run can be rerun and the variance checked. Shown as text — do not run live against a shared box.

Keywords — tap to unfold the plain meaning

Check yourself

Why is "our server does 10,000 tokens/s" not a benchmark on its own?
Which phase do input tokens stress, and which phase do output tokens stress?
How can a closed-loop client hide server overload — and what's that effect called?
What does goodput count that raw throughput does not?
Why are the cluster's 3,288 prompt tok/s and ~108 generation tok/s only "directional evidence"?

Hold workload and quality constant, change one variable, repeat, report variance, and keep the commands plus raw results. One lever per experiment, always reproducible.