Two ceilings, one plot
A GPU has two hard limits at once: how fast it can move bytes, and how fast it can crunch numbers. The roofline draws both on one chart so you can see, at a glance, which limit is actually in your way.
The slope and the roof
The roofline plots achievable performance against arithmetic intensity (FLOPs done per byte moved). On the left it's a rising bandwidth-limited slope; on the right it flattens into a compute-limited roof. Your kernel can never sit above either line.
Memory-bound vs compute-bound
If your arithmetic intensity is below the crossover, you're memory-bound — starved for bandwidth, with spare compute idling. Above it, you're compute-bound — the FLOPs are the limit and adding bandwidth wouldn't help.
Why it matters for decode
Decode re-reads the weights and KV for every single token, doing very few FLOPs per byte. That puts it far down the slope — deeply memory-bound — so the lever that actually helps is bandwidth and KV memory, never more raw FLOPs.
Keywords — tap to unfold the plain meaning
The roofline has a bandwidth slope and a compute roof. Below the crossover you're memory-bound; above it, compute-bound. Decode lives far down the slope.
Where's the line? The ridge point
"Which side am I on?" has a precise answer. Divide the GPU's peak compute by its memory bandwidth and you get the exact intensity where the slope meets the roof.
One number, two regimes
The ridge point is the arithmetic intensity where the bandwidth slope meets the compute roof. Below it, intensity < ridge means memory-bound. Above it, intensity > ridge means compute-bound. That's the whole decision.
Your H100 NVL numbers
An H100 NVL moves about 3.9 TB/s of memory and does roughly 836 TFLOPS dense in BF16 (~1,670 TFLOPS in FP8). Divide and the ridge sits near 214 FLOP/byte in BF16, 428 FLOP/byte in FP8.
Keywords — tap to unfold the plain meaning
Math, decoded
- peak FLOPsthe GPU's top compute rate — e.g. ~836 TFLOPS dense in BF16 on an H100 NVL
- memory bandwidthhow fast it can read/write memory — ~3.9 TB/s on the H100 NVL
- ridge pointthe FLOP-per-byte where the slope meets the roof: ~214 (BF16), ~428 (FP8)
Compare your kernel's arithmetic intensity to the ridge. intensity < ridge → memory-bound (you're on the bandwidth slope); intensity > ridge → compute-bound (you're under the roof).
3.9 TB/s bandwidth, ~836 TFLOPS BF16 dense, ~1,670 TFLOPS FP8. Ridge point ≈ 214 FLOP/byte in BF16 and ~428 FLOP/byte in FP8 — anything doing fewer FLOPs per byte than that is bandwidth-limited.
Ridge point = peak FLOPs ÷ memory bandwidth. On the H100s that's ~214 FLOP/byte (BF16) or ~428 (FP8). Below it, memory-bound; above it, compute-bound.
Batching slides the FC layers up
Weights get read once and reused by everyone in the batch. So the more requests you stack together, the more compute you squeeze out of every byte you hauled in — the FC layers climb the slope.
The lever: reuse per loaded weight
For a fully-connected (FC) layer, intensity is roughly 2 × batch ÷ bytes_per_weight. Each weight is loaded once and multiplied across all batch sequences, so raising the batch raises the FLOPs-per-byte — sliding that layer rightward up the roofline.
How far you'd have to push
At FP8 (1 byte per weight), batch 8 gives 2 × 8 ÷ 1 = 16 FLOP/byte. To actually reach the FP8/BF16 ridge you'd need roughly batch 214 — far past what a small decode batch ever reaches.
Keywords — tap to unfold the plain meaning
Math, decoded
- 2 ×a multiply-add is two FLOPs per weight per sequence
- batchhow many sequences share each loaded weight — the reuse factor
- bytes_per_weightstorage per weight: 1 for FP8, 2 for BF16
At batch 8, FP8: 2 × 8 ÷ 1 = 16 FLOP/byte — still deep on the slope. Reaching the ~214 ridge needs roughly batch 214.
FC intensity ≈ 2 × batch ÷ bytes-per-weight. Bigger batches reuse each weight more, climbing the slope — but reaching the ridge needs roughly batch 214.
Attention stays stuck on the slope
The FC trick relies on everyone sharing the same weights. Attention has no such shared object: every sequence drags its own KV cache, so there's nothing to reuse across the batch.
No cross-batch reuse
Each sequence keeps its own KV, so attention gets no cross-batch reuse. Where FC layers slide up the slope with batch, attention's per-sequence KV reads stay flat on the bandwidth slope — adding more sequences just adds more independent KV to re-read.
The split picture
So a decode step is two things at once: shared FC prep that can climb with batch, and per-sequence attention that stays memory-bound. The attention floor is why decode never escapes bandwidth, even when batching helps the FC layers.
Keywords — tap to unfold the plain meaning
Batching slides FC layers up the slope, but attention keeps each sequence's own KV — no cross-batch reuse, so it stays bandwidth-limited.
So where does decode actually sit?
Now plot the real model. At a typical decode batch, intensity is a tiny fraction of the ridge — so all that spare compute just sits there while bandwidth does the throttling.
Decode is nowhere near the roof
More GPU compute won't speed up decode because decode sits far below the roofline's ridge point. It's bottlenecked on memory bandwidth — re-reading weights + KV per token — not FLOPs, so spare compute goes unused.
The central punchline
You run out of memory long before compute becomes the limit. The lever that actually matters for decode is bandwidth + KV memory, not FLOPs. That reframes every optimization that follows.
What works, what's futile
What helps: quantization (fewer bytes per weight), GQA (a smaller KV footprint), and a smaller KV cache. What doesn't: adding FLOPs — futile, because the compute already sits idle.
Keywords — tap to unfold the plain meaning
--max-num-seqs 8: FC intensity = 2 × 8 ÷ 1 (FP8 weights) = 16 FLOP/byte ≈ 4% of ridge — deeply memory-bound. Generation throughput ~108 tok/s; prefill throughput 3,288 tok/s. Reaching the ridge would need ~batch 214.
Decode runs at ~4% of the ridge — deeply memory-bound. The levers that matter are quantization, GQA, and smaller KV; adding FLOPs is futile.
On your cluster · check yourself
You don't have to trust the chart. The same machine that gives you the ridge numbers gives you the throughputs — and the prefill-vs-decode gap is exactly the story the roofline predicts.
See the gap for yourself
Benchmark generation and prefill separately. Prefill processes the whole prompt in parallel — lots of FLOPs per byte — so it flies. Decode emits one token at a time, re-reading weights + KV, so it crawls. The numbers below are from the box.
Measured on the 4×H100 box · Qwen36 · FP8
vllm bench throughput \ --model Qwen36-FP8 \ --max-num-seqs 8 # prefill throughput .... 3,288 tok/s (compute-rich) # generation throughput .. ~108 tok/s (bandwidth-starved) # FC intensity @ batch 8 .. 16 FLOP/byte (~4% of ridge) # ridge point (FP8) ...... ~428 FLOP/byte (would need ~batch 214)
~4% of ridge — confirming it's memory-bound, not compute-bound. Prefill at 3,288 tok/s vs generation at ~108 tok/s is the roofline in one measurement: prefill is up near the roof, decode is far down the slope.
Keywords — tap to unfold the plain meaning
Check yourself · recall, don't peek
- Picture the slope, the roof, and the ridge — what does each one mean?
- What single formula gives the ridge point, and what's its value on your H100s in BF16?
- Why does batching lift the FC layers up the slope but leave attention stuck?
- Decode sits at ~4% of ridge — so which levers actually speed it up, and which is futile?
Prefill 3,288 tok/s vs decode ~108 tok/s is the roofline made real: compute-rich prefill near the roof, bandwidth-starved decode far down the slope.