Inference Engineering · Lesson 16 · Benchmarking an Inference Server CorrectlyHome · Glossary · Your Lab
Benchmarking an Inference Server Correctly
A number without a workload is not a benchmark.
Commit a prediction before revealing the model.
Today's win: design a reproducible inference benchmark and reject comparisons that mix workload shape, load model, SLO, or measurement boundaries.
The setup
You cannot compare kitchens by saying one served 2,000 items. Were they appetizers or banquets? One order at a time or a dinner rush? A benchmark fixes the menu, arrivals, and service promise before timing.
1 · Write the workload contract first
Record model and engine revisions, hardware, precision, prompt/output length distributions, sampling, cache state, concurrency or arrival rate, and streaming mode. Input and output tokens stress different phases.
2 · Warm up deliberately
First runs include model loading, compilation, allocator growth, and cold caches. Separate startup/cold-start tests from steady-state serving tests; do not silently discard whichever behavior matters to users.
3 · Choose closed-loop or open-loop load
Closed-loop clients send a new request after one finishes and can hide overload through coordinated omission. Open-loop arrivals follow an external rate and expose queues and tail latency. State which one you used.
4 · Report distributions and goodput
Report TTFT, TPOT/ITL, end-to-end latency, input/output throughput, queue time, errors, and p50/p95/p99. Goodput counts only requests meeting the SLO.
5 · Change one variable
Hold workload and quality constant, repeat runs, report variance, and retain commands plus raw results. Peak tokens/s without latency and quality constraints is not a serving result.
On YOUR cluster live-tested · course lab
The stored 3,288 prompt tok/s and ~108 generation tok/s values came from a live cluster run whose exact revisions and token lengths were not retained. This lesson explains why they are directional evidence and states exactly what the next run must record.
Today's win: design a reproducible inference benchmark and reject comparisons that mix workload shape, load model, SLO, or measurement boundaries.
The picture
You cannot compare kitchens by saying one served 2,000 items. Were they appetizers or banquets? One order at a time or a dinner rush? A benchmark fixes the menu, arrivals, and service promise before timing.
Write the workload contract first
Record model and engine revisions, hardware, precision, prompt/output length distributions, sampling, cache state, concurrency or arrival rate, and streaming mode
Warm up deliberately
First runs include model loading, compilation, allocator growth, and cold caches
Choose closed-loop or open-loop load
Closed-loop clients send a new request after one finishes and can hide overload through coordinated omission
Report distributions and goodput
Report TTFT, TPOT/ITL, end-to-end latency, input/output throughput, queue time, errors, and p50/p95/p99
1 · Write the workload contract first
Record model and engine revisions, hardware, precision, prompt/output length distributions, sampling, cache state, concurrency or arrival rate, and streaming mode. Input and output tokens stress different phases.
Notice the long right tail: p50 looks fine while p99 blows past the SLO, so goodput (requests that meet the SLO), not peak tokens/s, is the serving result. The load model matters too: closed-loop clients can hide overload through coordinated omission, while open-loop arrivals expose queues and tail latency.
2 · Warm up deliberately
First runs include model loading, compilation, allocator growth, and cold caches. Separate startup/cold-start tests from steady-state serving tests; do not silently discard whichever behavior matters to users.
Notice serving performance is conditional on every line of this contract, not just the GPU name. This matters because input-heavy and output-heavy workloads exercise prefill versus decode differently, so two tokens/s numbers from different contracts cannot be ranked, no matter how carefully each was measured.
3 · Choose closed-loop or open-loop load
Closed-loop clients send a new request after one finishes and can hide overload through coordinated omission. Open-loop arrivals follow an external rate and expose queues and tail latency. State which one you used.
Notice the early spike: model load, JIT/compile, allocator growth, and cold caches inflate the first requests. This matters because averaging them into a steady-state number understates serving latency, while discarding them hides real cold-start cost. Both are legitimate results, but they are different tests and must be labelled.
4 · Report distributions and goodput
Report TTFT, TPOT/ITL, end-to-end latency, input/output throughput, queue time, errors, and p50/p95/p99. Goodput counts only requests meeting the SLO.
Notice the feedback difference: a closed-loop client throttles itself when the server slows, so it cannot generate the overload it is supposed to measure (coordinated omission). This matters because capacity and tail-latency claims must come from open-loop load; closed-loop is fine for single-request latency, not for saturation.
5 · Change one variable
Hold workload and quality constant, repeat runs, report variance, and retain commands plus raw results. Peak tokens/s without latency and quality constraints is not a serving result.
Notice the only difference that makes a number comparable is discipline: fix workload, SLO, and quality, vary one thing, repeat, and keep the raw data. This matters because peak tokens/s with no latency or quality constraint is an unfalsifiable claim; it cannot be reproduced or used to choose between engines.
Try it: open-loop vs closed-loop, live
This is that distribution, live. Push offered load toward 100% and watch the tail blow past the SLO; then switch to closed-loop and watch the same overload hide itself: coordinated omission in one click.
On YOUR cluster live-tested · course lab
The stored 3,288 prompt tok/s and ~108 generation tok/s values came from a live cluster run whose exact revisions and token lengths were not retained. This lesson explains why they are directional evidence and states exactly what the next run must record.