Model Parallelism & NVLink — Inference Engineering

01

The setup: one model, N GPUs

TL;DR · When one model is too big for a single GPU, you spread it across several — but then the GPUs must talk to each other constantly, and that conversation is the whole story.

Up to now a model has lived on one GPU. This part of the course asks what changes when it can't — and the surprising answer is that the wire between the GPUs, not the GPUs themselves, decides whether splitting was worth it.

Two ways to use more than one GPU

You can run replicas — put a full copy of the model on each GPU and send different requests to each. Or you can run model parallelism — split one copy of the model across several GPUs so they cooperate on every request. This lesson is about the second.

Tensor parallelism is the workhorse

The form of model parallelism used for inference is tensor parallelism (TP): run one model across N GPUs by sharding each layer's weight matrices. Each GPU holds a slice of the weights and computes a partial result.

The catch arrives immediately

Because each GPU only has a slice, the partial results must be stitched back together before the next layer. That stitching is communication over a wire — and on inference's per-token critical path, the speed of that wire is what makes or breaks the whole idea.

Keywords — tap to unfold the plain meaning

Analogy ❝TP is one dish, two chefs. Chef A preps the left half, chef B the right; they merge into one plate before the next step. Replicas are the opposite: two chefs each cooking a whole separate order, never needing to talk. The moment you make two chefs share one dish, the speed of the window between their stations starts to matter more than how fast either chef chops.

Replicas duplicate the whole model; tensor parallelism splits one model across GPUs. Splitting buys you capacity but forces the GPUs to talk — and the wire becomes the bottleneck.

02

What tensor parallelism actually splits

TL;DR · TP shards each layer's weight matrices across GPUs — column-parallel then row-parallel — so every GPU computes a partial result, and an all-reduce merges them before the next layer.

"Split the model" is vague. The precise version: every big matrix in every layer is cut into slices, one slice per GPU. Knowing exactly where the cut happens is what tells you where the communication has to happen.

Slice the weight matrices

TP takes each layer's weight matrices and shards them across the N GPUs. A column-parallel layer splits the weights by columns; the following row-parallel layer splits by rows. Each GPU multiplies only its slice and produces a partial output.

Partial results, then a merge

No single GPU has the full answer for a layer — each holds one piece. Before the next layer can run, the pieces are combined with an all-reduce: every GPU sends its partial result, and they all come away with the summed-up full result.

This is Megatron's design

This column-parallel / row-parallel + all-reduce pattern is the original recipe from Megatron-LM (Shoeybi et al., 2019, arXiv 1909.08053) — still the template every inference engine uses for TP today.

Each GPU multiplies only its weight slice and gets a partial result; the all-reduce in the middle is the communication step that stitches the partials into the full result the next layer needs.

Keywords — tap to unfold the plain meaning

TP shards each layer's matrices (column-parallel, then row-parallel) so every GPU computes a partial result, and an all-reduce merges them before the next layer. That's Megatron's recipe.

03

The hidden cost: all-reduce every layer, every token

TL;DR · Megatron does two all-reduces per transformer layer, so a 64-layer model fires ~128 all-reduces to emit one token — all on the critical path, where the next layer waits.

One all-reduce sounds cheap. The problem is how many of them there are, and that none can be hidden: each is a hard stop where every GPU waits for everyone else before the next layer can start.

Two all-reduces per layer

Megatron does two all-reduces per transformer layer — one after attention, one after the MLP. Each is a point where the GPUs must synchronize and exchange their partial results.

Multiply by depth, per token

In decode, a 64-layer model therefore needs ~128 all-reduces to emit one token. And it pays that toll for every single generated token, not once per request.

All on the critical path

None of this overlaps useful work: the all-reduces sit on the critical path — the next layer waits until the merge finishes. Slow communication doesn't just add overhead; it directly stretches per-token latency.

Math, decoded

all-reduces / token = 2 × L → 2 × 64 = 128

Lthe number of transformer layers in the model — 64 in this example
2 ×two all-reduces per layer: one after attention, one after the MLP
128total all-reduces to produce a single token — and it repeats for every token

Two all-reduces per layer, times 64 layers, is ~128 synchronizations on the critical path for each token you generate. The deeper the model, the more often the GPUs must stop and talk.

Keywords — tap to unfold the plain meaning

Analogy ❝Now the two chefs hand off on every step of every dish. A fast window between stations means the handoffs are seamless and you barely notice them. A slow hallway means they're walking back and forth all night — and the food, however fast they chop, comes out late. With ~128 handoffs per plated bite, the hallway is the meal.

Two all-reduces per layer × 64 layers = ~128 synchronizations to emit one token, all on the critical path. The next layer can't start until the merge finishes.

04

So the wire matters: NVLink vs PCIe

TL;DR · Those ~128 all-reduces per token ride a physical wire. On NVLink (~640–900 GB/s) it's fine; on PCIe (~128 GB/s, ~7× slower) the same TP becomes a latency disaster.

If communication is on the critical path and happens ~128 times per token, then the bandwidth of the link between GPUs is no longer a footnote — it's the single number that decides whether tensor parallelism helps or hurts.

NVLink is the fast lane

NVLink is NVIDIA's direct GPU-to-GPU interconnect, running roughly 640–900 GB/s. On NVLink, the all-reduces are quick enough that the merge barely shows up in per-token latency.

PCIe is the slow hallway

PCIe — the general-purpose bus — moves data at only ~128 GB/s, about 7× slower than NVLink. Force ~128 all-reduces per token across PCIe and each one becomes a visible stall.

Same model, very different result

Identical TP setup, identical model: on NVLink it's a win, on PCIe it's the worst case. The compute didn't change — only the wire did — yet the wire is what governs the outcome.

Numbers, decoded

NVLink ≈ 640–900 GB/s vs PCIe ≈ 128 GB/s → ~7× slower

GB/sgigabytes per second — how fast partial results can move between GPUs
640–900NVLink's range; fast enough that all-reduce barely dents per-token latency
128PCIe's bandwidth; the same all-reduces now stall on a slow bus
~7×how much slower PCIe is — multiplied across ~128 all-reduces per token

The merge step is communication-bound, so its time scales with bandwidth. Dropping from ~640–900 GB/s to ~128 GB/s makes every one of the ~128 per-token all-reduces ~7× slower.

Keywords — tap to unfold the plain meaning

Analogy ❝NVLink is a wide pass-through window right between the two chefs' stations — they hand the plate across in a blink. PCIe is making them walk down a long shared hallway for every handoff. With one handoff a night, who cares. With ~128 per bite, the hallway decides how fast the kitchen runs, no matter how sharp the knives.

NVLink ≈ 640–900 GB/s; PCIe ≈ 128 GB/s, ~7× slower. Same model, same TP — the wire alone decides whether the all-reduces are invisible or crippling.

05

TP=N or replicas? The decision

TL;DR · If the model fits on one GPU, replicas (TP=1 × N) usually win on throughput. If it doesn't fit — or you need minimum token latency — use TP=N, but only when it's on NVLink.

Because TP's cost is all that communication, the right question is never "how many GPUs?" but "do I actually need to split, and is the wire fast enough to make splitting pay?"

Fits on one GPU → replicas

If the model fits in one GPU's memory, replicas (TP=1 × N) usually win on throughput. Each GPU runs a full copy independently with zero all-reduce overhead — no GPU ever waits on another.

Doesn't fit, or latency-critical → TP=N

If the model doesn't fit, or you need minimum token latency, use TP=N — but it's only worth it on NVLink. TP can lower latency by spreading one token's compute over N GPUs, provided the wire doesn't eat the savings.

The trade is throughput vs latency

Replicas maximize total tokens served (throughput) by avoiding communication. TP minimizes time-per-token (latency) by parallelizing one request — at the cost of the all-reduce tax. NVLink is what keeps that tax small enough to be worth paying.

First ask if the model fits on one GPU and your latency is fine — if so, replicas win. Only split into TP=N when you must, and only then if the GPUs are on NVLink.

Keywords — tap to unfold the plain meaning

Fits on one GPU → replicas (TP=1 × N) win throughput. Doesn't fit, or you need minimum token latency → TP=N — but only worth it on NVLink.

06

On your cluster: this is exactly the trap

TL;DR · The 4×H100 box runs a model that fits on one GPU as TP=2 — across the one GPU pair whose NVLink is down — so ~128 all-reduces per token crawl over PCIe. It's the worst case, and avoidable.

Every rule in this lesson collides in one real misconfiguration on the cluster. A model that needed no splitting is split, onto the one link that can't carry it. Seeing it makes the whole lesson click.

The fault, exactly

On the cluster, qwen35-27b (27B, FP8, ~27 GB) runs with --tensor-parallel-size 2 on GPU0 + GPU1 — the one pair whose NVLink is down. So the ~128 all-reduces per token cross PCIe (~128 GB/s): the worst case from Station 04.

Why it's doubly wrong

The model is ~27 GB — it fits comfortably on a single 94 GB H100. By the decision tree, it should run as a replica with no all-reduce at all. Instead TP=2 was chosen, adding communication that creates latency rather than reducing it — and then routed over the slow wire on top.

The misconfiguration on the 4×H100 box

vllm serve qwen35-27b \
  --tensor-parallel-size 2     # GPU0 + GPU1
  # ↳ but NVLink on this pair is DOWN
  # ↳ ~128 all-reduces/token now cross PCIe (~128 GB/s)
  # ↳ and the model (~27 GB) fits on ONE 94 GB H100 anyway

Cluster note 4×H100 finding. qwen35-27b (27B FP8, ~27 GB) on --tensor-parallel-size 2 over GPU0 + GPU1, whose NVLink is down — forcing all-reduces across PCIe (~128 GB/s, ~7× slower than NVLink's 640–900 GB/s). Since it fits on one 94 GB GPU, TP=2 is unnecessary: it manufactures latency instead of reducing it. Fix: run it as a single-GPU replica, or pin TP to an NVLink-connected pair.

Keywords — tap to unfold the plain meaning

Check yourself (recall, don't peek)

Picture the two chefs and the combine step — what exactly does tensor parallelism split, and what stitches the slices back together?
Why does a 64-layer model fire ~128 all-reduces to emit one token, and why can't that work be hidden?
NVLink vs PCIe — why does the same TP setup win on one wire and fail on the other?
The model fits on a single GPU. Should you run TP=2 or a replica, and why?

A model that fits on one GPU, split TP=2 across a dead-NVLink pair, sends ~128 all-reduces/token over PCIe. It's the worst case — and the fix is simply a replica.