GPU Generations — Inference Engineering

01

Three axes move each generation

TL;DR · NVIDIA ships a new GPU generation every couple of years; for inference, only three things really move — precision, interconnect, and memory.

Ada → Hopper → Blackwell → Rubin sounds like a lot to track. It isn't. Lock these three axes in your head and every spec sheet becomes legible.

Three things improve each gen

Generation to generation, the things that matter for inference are: precision (new low-bit formats with hardware support), interconnect (how fast GPUs talk, decisive for tensor parallelism and disaggregation), and memory (capacity + bandwidth, which set your KV ceiling and roofline).

The lineup, one row each

Ada (RTX 40xx): FP8, no NVLink. Hopper (H100): FP8 · NVLink 4 · HBM3. Blackwell (B200/GB200): FP4 · NVLink 5 · HBM3e. Rubin: next — precision down, interconnect up, memory up. The trend is always the same direction.

Keywords — tap to unfold the plain meaning

Each generation lowers the minimum number format, widens the GPU-to-GPU links, and grows memory. Your 4×H100s sit at the Hopper column.

Analogy ❝Each GPU generation is a newer kitchen build. The ovens gain a coarser-but-faster setting (lower precision — FP8, then FP4). The delivery roads between cooking stations get wider (faster NVLink). And the pantries get bigger and closer (more, faster HBM). Same kitchen idea, just a better-equipped model year.

Each generation moves three axes for inference: precision (lower bits), interconnect (faster NVLink), and memory (more, faster HBM). Everything else is detail.

02

Precision: FP16 → FP8 → FP4

TL;DR · Each generation adds hardware support for a lower-bit number format — FP8 arrived on Hopper, FP4 on Blackwell — so the same silicon does more math per second.

"Lower precision" sounds like losing something. On these chips it's the opposite: a coarser-but-faster oven setting that the Tensor Cores are built to run flat-out.

New low-bit formats, in hardware

The first axis is precision: each generation introduces new low-bit formats with real hardware support — FP8 on Hopper, FP4 on Blackwell. Hopper introduced FP8 Tensor Cores plus the Transformer Engine to drive them.

Why it matters for inference

Hardware FP8/FP4 is what makes a quantized model fast rather than just small. An FP8 model like your Qwen runs best where FP8 Tensor Cores exist — which is exactly the workload Hopper was built for.

Keywords — tap to unfold the plain meaning

The precision ladder, decoded

FP16 (16 bits) → FP8 (8 bits, Hopper) → FP4 (4 bits, Blackwell)

FP1616-bit floating point — the older high-precision baseline for weights and activations
FP88-bit format with Hopper Tensor Core support — half the bits, roughly double the math throughput
FP44-bit format with Blackwell hardware support — for frontier-size models that can tolerate it

Fewer bits per number means more numbers moved and multiplied per second — but only when the Tensor Cores natively support that format. That hardware support is the real generation jump.

Precision falls each generation with hardware behind it: FP8 on Hopper, FP4 on Blackwell. Match your model's format to the chip that supports it.

03

Interconnect: faster NVLink & bigger domains

TL;DR · The second axis is how fast GPUs talk — NVLink 4 → 5 and wider NVSwitch domains — which is decisive for tensor parallelism and disaggregated serving.

A faster oven doesn't help if the delivery road between stations is a single lane. Interconnect is that road, and each generation widens it.

Faster links, wider switch fabric

The second axis is interconnect: faster NVLink/NVSwitch. Hopper carries NVLink 4; Blackwell moves to NVLink 5. This is decisive for tensor parallelism and disaggregation, where GPUs swap data constantly.

One domain that acts like one machine

The GB200 NVL72 wires 72 GPUs into a single NVLink domain, so giant models and disaggregated serving behave almost like one machine. Bigger domains are the unlock for frontier-scale deployments.

Keywords — tap to unfold the plain meaning

Analogy ❝Picture each GPU as a cooking station. Tensor parallelism makes them share one dish, passing half-finished sauces back and forth nonstop. If there's only one narrow road between stations, the cooks wait on each other. A bigger NVLink domain is paving every road wide enough that 72 stations feel like one open kitchen — handoffs stop being the bottleneck.

Interconnect (NVLink 4 → 5, NVSwitch domains) decides how well GPUs cooperate. The GB200 NVL72 fuses 72 GPUs into one domain for frontier-scale serving.

04

Memory: HBM3 → HBM3e

TL;DR · The third axis is memory — more GB and more TB/s of HBM — which sets your KV-cache ceiling and where you land on the roofline.

Decode is memory-bandwidth bound: the faster and bigger the pantry, the more requests you can keep in flight before you run dry.

More capacity, more bandwidth

The third axis is memory: capacity plus bandwidth. Hopper carries HBM3; Blackwell upgrades to HBM3e — more GB and more TB/s. These set your KV ceiling (how many tokens you can cache) and your roofline (whether you're compute- or bandwidth-bound).

Hopper's number, concretely

Hopper's H100 delivers ~3.9 TB/s HBM3. That bandwidth is exactly what keeps an FP8 model's decode loop fed — the right pantry for the workload you run.

Keywords — tap to unfold the plain meaning

Analogy ❝Think of HBM as the pantry right beside the line cook. A bigger pantry (more GB) means more orders cached and ready; a faster pantry (more TB/s) means the cook grabs ingredients without sprinting to a faraway storeroom. Memory bandwidth is how fast that nearby pantry hands things over — and on the decode loop, it's usually the thing the cook is waiting on.

Memory (HBM3 → HBM3e, more GB and TB/s) sets your KV ceiling and roofline. Hopper's ~3.9 TB/s HBM3 is the pantry feeding your FP8 decode loop.

05

Choosing: match the generation to your workload

TL;DR · FP8 inference at normal scale → Hopper is excellent value. Frontier models, FP4, or huge multi-GPU domains → Blackwell. Don't overbuy capability you can't use.

The newest chip isn't the right chip — the matching chip is. Buying Blackwell to run an FP8 model at normal scale is paying for ovens you'll never set to FP4.

The rule of thumb

FP8 inference at normal scale → Hopper is excellent value. Frontier models, FP4, or huge multi-GPU domains → Blackwell. The decision is just: does your model and traffic actually use the newer capability?

When Blackwell earns its price

You'd move to Blackwell only to (a) run much larger models, (b) exploit hardware FP4, or (c) build large NVLink domains for disaggregated serving. Short of that, it's overbuying.

Rubin is next

Rubin continues the same trend — precision down, interconnect up, memory up — and is the tier above Blackwell for the largest deployments. The choosing logic doesn't change: match the axis you actually use.

Keywords — tap to unfold the plain meaning

Analogy ❝If your menu only ever needs the FP8 oven setting, buying the kitchen with the FP4 super-oven and the 72-station open floor is paying for equipment that sits cold. The right model year is the cheapest one whose ovens, roads, and pantry your menu actually fills — not the flashiest one in the catalogue.

FP8 at normal scale → Hopper is the value sweet spot. Frontier size, FP4, or huge NVLink domains → Blackwell. Don't overbuy capability your model and traffic can't use.

06

On your cluster

TL;DR · Your 4× H100 NVL (Hopper) are a great match for FP8 Qwen serving — you're well-matched and not leaving much on the table.

This isn't abstract spec-comparing. For the exact model you serve, the chip in your rack is the value sweet spot — here's why, in one sentence you can teach back.

Well-matched, by the numbers

Your 4× H100 NVL (Hopper) are a great fit for FP8 Qwen serving: the model is FP8, and Hopper's FP8 Tensor Cores + ~3.9 TB/s HBM3 are precisely what that needs. It's the value sweet spot for inference at your scale.

When you'd reach for Blackwell

You'd jump to Blackwell only to (a) run much larger models, (b) exploit FP4, or (c) build big NVLink domains for disaggregated serving. For today's workload, you're not leaving much on the table.

Cluster note On 4× H100 NVL (Hopper): a great match for FP8 Qwen, with FP8 Tensor Cores + ~3.9 TB/s HBM3 and NVLink 4. Teach it back as: "We chose H100s because Hopper's FP8 Tensor Cores + fast HBM3 are exactly what an FP8 model like our Qwen needs — the value sweet spot for inference at our scale." Jump to Blackwell only for frontier-size models, FP4, or big NVLink domains for disaggregated serving; otherwise it's overbuying.

Keywords — tap to unfold the plain meaning

Runnable companion (lab notebook reference)

# day20 notebook: gpu-generations-hopper-blackwell.ipynb
# compares Hopper vs Blackwell for an FP8 Qwen workload

Check yourself

Name the three axes that improve each GPU generation.
Which generation introduced hardware FP8, and which introduced FP4?
What does the GB200 NVL72 do, and why does it matter for disaggregated serving?
Finish the sentence: "We chose H100s because…" — and name the three reasons you'd instead move to Blackwell.

Your 4×H100 (Hopper) are well-matched to FP8 Qwen: FP8 Tensor Cores + ~3.9 TB/s HBM3. Blackwell only earns its price for bigger models, FP4, or large NVLink domains.