Disaggregated Serving — Inference Engineering

01

The two halves fight on shared hardware

TL;DR · Prefill saturates compute; decode saturates memory bandwidth. Run both on the same GPU and a long prefill stalls everyone's decode.

You already know a request has two phases. The trouble is they want opposite things from the hardware — and when they share a GPU, the slow one drags the fast one down.

Two phases, two bottlenecks

Prefill reads the whole prompt in one big parallel pass — it's compute-bound, it saturates the math units. Decode emits one token at a time, re-reading the KV cache each step — it's memory-bound, it saturates memory bandwidth. Same model, opposite pressure points.

Sharing creates head-of-line stalls

On shared hardware, a long prefill grabs the compute and a long prefill stalls everyone's decode. The steady token stream other users are waiting on hitches every time a big prompt lands. One slow job blocks the fast lane.

Keywords — tap to unfold the plain meaning

Analogy ❝Picture one kitchen doing everything. The prep cooks are heavy and bursty — they chop a mountain of vegetables for one giant order and seize every counter. Meanwhile the plating line, which should be steady and fast, can't get a single dish out because the preppers have taken all the space. One burst of heavy prep work, and every quick plate behind it waits. The two jobs want different things from the same room.

Prefill is compute-bound, decode is memory-bound. Share one GPU and a long prefill stalls everyone's decode — that stall is the whole reason to split them apart.

02

Split into two pools: xPyD

TL;DR · Disaggregated serving runs dedicated prefill workers and dedicated decode workers as separate pools — each tuned for its own bottleneck, each scaling independently.

The fix is simple to state: stop making one GPU do both jobs. Give prefill its own pool, give decode its own pool, and size each for the pressure it actually feels.

Dedicated pools, separately tuned

Disaggregated serving runs dedicated prefill workers and dedicated decode workers as separate pools. Each is tuned for its own bottleneck — compute vs. memory bandwidth — and each pool scales independently. If prefill is the queue that's backing up, add prefill workers without touching decode.

The xPyD notation

Configurations are written "xPyD": x prefill workers, y decode workers. A 2P4D deployment is two prefill workers feeding four decode workers — you tune the ratio to your traffic's prompt-length and output-length mix.

The prefill pool builds the KV cache, ships it across the fabric, and the decode pool streams the answer. "xPyD" just names how many of each you run.

Keywords — tap to unfold the plain meaning

Analogy ❝So split the kitchen in two. A separate prep kitchen — heavy and bursty — does nothing but chop and prep, with as many stations as the prep load demands. A separate plating line — steady and fast — does nothing but plate and serve. Between them, runners ship prepped trays across. Now a giant prep order can't freeze the plating line, because they're in different rooms, each staffed for its own kind of work.

Disaggregation = dedicated prefill workers + dedicated decode workers, scaling independently. Written xPyD: x prefill, y decode. The prep kitchen and the plating line, in separate rooms.

03

The catch: moving the KV cache

TL;DR · The prefill worker's KV cache must be shipped to the decode worker. If that transfer costs more than just decoding in place, disaggregation loses.

Splitting the kitchen isn't free. The prep work has to physically travel to the plating line — and if the trip is slow enough, you'd have been better off never splitting at all.

The handoff is the whole cost

The prefill worker's KV cache must be transferred to the decode worker, over NVLink, InfiniBand, or a layer like NIXL. That cache is exactly the per-prompt attention state decode needs to keep going — and it can be large.

The break-even rule

Here's the trade in one line: if that transfer costs more than just decoding in place, disaggregation loses. The whole technique only pays off when shipping the cache is cheaper than the head-of-line stalls you removed by splitting.

The break-even, decoded

disaggregate ⇔ t_transfer < t_stall

t_transfertime to ship the prompt's KV cache from a prefill worker to a decode worker across the fabric
t_stalldecode time you would have lost to head-of-line stalls if prefill and decode shared one GPU
⇔disaggregation only wins when the transfer is cheaper than the stall it removes; otherwise it loses

Splitting pays off only when moving the KV cache is faster than the decode time you were losing to shared-GPU stalls. A slow fabric flips the inequality and disaggregation loses.

Keywords — tap to unfold the plain meaning

Analogy ❝Now the prep kitchen is across town. Every order means loading prepped trays onto a truck and driving them to the plating line. If the road is a fast highway, fine — the trip is nothing. But if the prep kitchen is a faraway pantry down a single slow road, the truck ride eats up more time than you ever saved by separating the rooms. At that point you should have just kept one kitchen.

The KV cache must travel from prefill to decode over NVLink, InfiniBand, or NIXL. If that trip costs more than decoding in place, disaggregation loses. The fabric is everything.

04

When it wins — and who orchestrates it

TL;DR · Disaggregation wins at large scale — many GPUs, long prompts, a fast fabric. NVIDIA Dynamo orchestrates it on systems like the GB200 NVL72.

The break-even rule tells you exactly where this technique belongs: not on a small box, but where the GPUs are plentiful, the prompts are long, and the wires between them are very, very fast.

The payoff conditions

Disaggregation wins at large scale: many GPUs, long prompts, and a fast fabric. Long prompts make prefill heavy enough that isolating it really helps; many GPUs give you room to size pools independently; a fast fabric keeps the KV-cache transfer below break-even.

The orchestrator

That's exactly what NVIDIA Dynamo orchestrates — dynamic prefill/decode pools and KV routing — on systems like the GB200 NVL72 (72 GPUs as one NVLink domain). One giant NVLink domain is the fast fabric that makes the cache handoff nearly free.

Cluster note On your 4×H100 box, you currently co-locate prefill and decode and soften the head-of-line stalls with --enable-chunked-prefill (from Lesson 18), which interleaves prefill into decode steps. Disaggregation is the next tier — it needs a faster interconnect and larger scale than four GPUs to clear break-even. The research baseline is DistServe (Zhong et al., arXiv:2401.09670), which introduced disaggregating prefill and decoding.

Keywords — tap to unfold the plain meaning

Win conditions: many GPUs, long prompts, a fast fabric. NVIDIA Dynamo orchestrates dynamic pools + KV routing on the GB200 NVL72's 72-GPU NVLink domain. On 4×H100, chunked prefill is the right tool for now.

05

On your cluster

TL;DR · You're below the disaggregation threshold — so co-locate with chunked prefill today, and keep Dynamo in your back pocket for when scale and fabric grow.

The point isn't to disaggregate everything. It's to know which tier you're on. On four GPUs the answer is "not yet" — and that's a real engineering decision, not a shortcoming.

Today: co-locate and chunk

Keep prefill and decode on the same workers and run --enable-chunked-prefill. Chunking breaks a long prefill into smaller pieces interleaved with decode steps, so a big prompt no longer freezes the token stream — the cheap fix for a single-box stall.

Later: the next tier

When you grow to many GPUs on a fast fabric, revisit disaggregation. The runnable references to study: day12 nvidia-dynamo.ipynb and day18 disaggregation-prefill-decode.ipynb.

Keywords — tap to unfold the plain meaning

Today's choice on the 4×H100 box — co-locate, soften with chunking

vllm serve Qwen3.6-27B-FP8 \
  --tensor-parallel-size 4 \
  --enable-chunked-prefill

Check yourself

Explain to a colleague: "Disaggregation helps when…" (name the three win conditions and the one transfer cost that can sink it).
Why is prefill compute-bound and decode memory-bound — and how does that cause one to stall the other on a shared GPU?
What does "2P4D" mean, and what single cost decides whether splitting into those pools is worth it?

Right tier, right tool: on 4×H100, co-locate with --enable-chunked-prefill. Disaggregation is the next tier up, once you have many GPUs and a fast fabric.