Prefill vs Decode — Inference Engineering

01

One model, two speeds

TL;DR · Every request splits into two phases — prefill (read the prompt) and decode (write the answer) — and on real hardware decode is about 30× slower.

The same weights, the same GPU, the same request — yet reading the prompt screams and writing the reply crawls. That one gap explains nearly every serving decision you'll make later.

The loop has two halves

An inference request runs in two phases. Prefill reads the whole prompt in one pass and emits the first token. Decode then emits the rest of the tokens one at a time, each step feeding back the token it just made.

The measured gap is huge

On the cluster, a 27B model hits ~3,288 tokens/sec during prefill but only ~108 tokens/sec during decode — roughly 30× slower. Same model, same chips. The phase is what changed.

This asymmetry is the whole lesson

Prefill and decode aren't just two steps; they stress two different parts of the GPU. Understanding which part each one hits is the key that unlocks batching, the KV cache, and disaggregated serving down the line.

Measured on the cluster — Qwen3.5-27B on 4×H100, same model, same request, two phases:

prefill (read)3288

decode (write)108

Tokens per second. Reading the prompt is ~30× faster than writing the answer — the central asymmetry of this lesson.

Keywords — tap to unfold the plain meaning

Every request = prefill (read the prompt, one pass) + decode (write the answer, one token at a time). Decode is ~30× slower on real hardware.

02

Prefill: read the whole prompt at once

TL;DR · Prefill processes every prompt token together in one big matrix multiply, so it saturates the tensor cores — it's compute-bound.

Hundreds or thousands of prompt tokens arrive at the same time. The GPU loves that: one enormous multiply keeps every tensor core busy, and the bytes you moved get reused across the whole batch of positions.

All tokens, one pass

Prefill reads the whole prompt in a single forward pass and produces the first output token. Because every prompt position is available at once, it becomes one large matrix multiply rather than many tiny ones.

It saturates the math units

Big matrix multiplies keep the tensor cores fully fed, so prefill is compute-bound — limited by raw FLOPs, the math throughput of the chip, not by how fast bytes move.

Prompts are usually the bulk

Typical RAG traffic carries a heavy prompt:generation ratio — measured at 19:1 to 47:1 on the cluster. Most of your tokens are read, not written, which is exactly why the fast phase matters.

Keywords — tap to unfold the plain meaning

Cluster note On the 4×H100 box, real RAG requests showed a prompt:generation ratio of 19:1 to 47:1 — for every token written, 19 to 47 were read first. That's why the fast, compute-bound prefill phase dominates total token counts even though decode dominates the clock.

Prefill = all prompt tokens in one big matrix multiply. It saturates the tensor cores, so it's compute-bound and fast.

03

Decode: one byte-heavy token at a time

TL;DR · Decode writes answers one token at a time, and each token re-reads all the model's weights from memory while doing only one token's worth of math.

Now the prompt's gone and the model must invent the answer token by token. Each step is tiny in math but enormous in bytes: the GPU drags the entire model out of memory just to produce a single next token.

Sequential, by nature

Decode emits tokens one at a time, each step feeding back the token it just made. You can't compute token n+1 until token n exists, so the steps can't be fused into one big multiply the way prefill can.

Every token re-reads the whole model

Each decode step re-reads all the weights — huge bytes pulled from HBM — but does only one token's worth of math. The KV cache it built during generation gets re-read too.

Lots of bytes, little math

That mismatch — gigabytes moved, a sliver of compute done — is the heart of why decode is slow. The chip's math units sit mostly idle, waiting on memory.

Keywords — tap to unfold the plain meaning

Analogy · the faraway pantry ❝Think of the kitchen this way: the pantry (HBM) holds all your ingredients — the model weights and the KV cache. The road and van between pantry and counter is your memory bandwidth. The stoves are compute, the FLOPs. Prefill is one big catering order, cooked in a single go — the stoves blaze. Decode is à la carte, one bite at a time: for every single bite, the van has to drive all the way to the faraway pantry and haul back the entire stock, just to cook one mouthful. The stoves barely fire; the van is the bottleneck.

Decode writes one token per step and re-reads every weight (huge bytes from HBM) for each — big on bytes, tiny on math.

04

Why decode is bandwidth-bound

TL;DR · Decode moves huge bytes but does little math, so its arithmetic intensity is tiny — memory bandwidth, not FLOPs, is the ceiling.

"Compute-bound" and "memory-bandwidth-bound" sound like jargon until you see the one ratio that decides which you are. Decode lands hard on the wrong side of it — and no amount of faster math helps.

The deciding ratio

Arithmetic intensity is FLOPs per byte moved. High intensity means lots of math per byte — you're compute-bound. Low intensity means lots of bytes per unit of math — you're memory-bandwidth-bound.

Decode's intensity is tiny

Each decode step moves all the weights but does one token's math, so FLOPs-per-byte is small. The bottleneck is memory bandwidth — how fast bytes move from HBM — not the tensor cores' speed.

Prefill sits on the other side

Prefill reuses each loaded byte across many prompt positions, so its arithmetic intensity is high and it's compute-bound. Same weights, opposite ceiling — purely because of how many tokens share each byte-read.

Math, decoded

arithmetic intensity = FLOPsbytes moved → decode: small · prefill: large

FLOPsfloating-point math operations done this step — the work the tensor cores perform
bytes movedweights (and KV cache) hauled out of HBM to do that work
ratio smalldecode: many bytes, one token's math → memory-bandwidth-bound
ratio largeprefill: each byte reused across many positions → compute-bound

Arithmetic intensity is FLOPs per byte. When it's small, the chip finishes its math long before the next bytes arrive, so memory bandwidth sets the speed. Decode lives here; faster math units wouldn't move the needle.

Prefill keeps the stoves (tensor cores) busy; decode keeps the van (memory bandwidth) busy. The ~30× gap is the ratio of those two ceilings on this model.

Keywords — tap to unfold the plain meaning

Arithmetic intensity = FLOPs ÷ bytes. Prefill's is high (compute-bound); decode's is tiny (bandwidth-bound). The ceiling, not the math, decides the speed.

05

Batching beats the asymmetry

TL;DR · One weight-read can serve a whole batch of decode requests, so batching multiplies decode throughput — until ~batch 32, where it turns compute-bound.

If decode wastes its expensive byte-read on a single token, the fix writes itself: make that one byte-read feed many tokens at once. That's batching, and it's why your serving engine never runs requests one by one.

Reuse the expensive read

Batching serves multiple sequences at the same time: one weight-read from HBM serves the whole batch. The bytes you were already paying for now produce many tokens instead of one.

Throughput climbs, then flips

More requests per weight-read means higher decode throughput. But past roughly batch 32, decode's matrix multiplies grow large enough to saturate the tensor cores — and decode becomes compute-bound, just like prefill.

Bigger answers need more GPUs

When one model is too large or the batch too heavy for a single GPU, tensor parallelism splits one model's layers across several GPUs — the 4×H100 setup here is exactly that. It spreads both the bytes and the math.

Keywords — tap to unfold the plain meaning

Cluster note On the 4×H100 box, decode's matrix multiplies become compute-bound past roughly batch 32. Below that, you're paying for bytes you're not fully using, so raising the batch size is almost free throughput; above it, you've hit the math ceiling and batching stops helping decode.

One weight-read serves the whole batch, so batching multiplies decode throughput — until ~batch 32, where decode turns compute-bound.

06

On your cluster

TL;DR · Run the cluster probe to measure prefill vs decode yourself — then explain the 30× gap out loud, in terms of the van and the pantry.

You don't have to trust the 3,288-vs-108 numbers — reproduce them. One script reads the two speeds straight off the box so the asymmetry stops being a slide and becomes your slide.

Measure both phases

The cluster probe runs a prompt through the server and reports the two throughputs separately — prefill tokens/sec and decode tokens/sec — so you can watch the ~30× gap appear on your own hardware.

Run it on the 4×H100 box

bash learning/tools/cluster-probe.sh

Cluster note Measured on the 4×H100 box with Qwen3.5-27B: prefill ~3,288 tok/s, decode ~108 tok/s — about 30× slower. The probe reuses tensor parallelism across all four GPUs, which is why these are the numbers you'll see, not single-GPU figures.

Keywords — tap to unfold the plain meaning

Check yourself

Finish this out loud to a colleague: "Our 27B is 30× slower at writing than reading because…"
Which phase is compute-bound and which is memory-bandwidth-bound — and why?
In the pantry analogy, what are the weights/KV cache, the road-and-van, and the stoves?
Why does batching speed up decode, and what happens past roughly batch 32?

Prefill reads fast because the stoves stay busy; decode writes slow because the van keeps hauling the whole pantry for one bite. That gap is your whole serving stack.