GPU Architecture: SMs & HBM — Inference Engineering

01

The kitchen building itself

TL;DR · A GPU is a kitchen with ~130 cook stations (SMs), each holding general tools (CUDA cores) and one specialized appliance (a Tensor Core). The pantry is HBM; each station's cutting board is tiny, instant SRAM.

We've talked about the pantry and the van for the whole course. Now look at the building. Once you see the floor plan, every optimization you've met — decode being slow, fusion, big batches — becomes obvious.

Three parts, one floor plan

A modern GPU is just three kinds of thing wired together: compute units (the SMs, where your code actually runs), one matrix-multiply engine per unit (the Tensor Core), and a memory hierarchy that trades capacity for speed. Learn those three and the chip stops being a black box.

This is Part V — The Hardware

Earlier parts treated the GPU as a fast box you feed work to. From here on we open the lid. This lesson is the floor plan; the next ones (GPU generations, MIG) build on it.

Keywords — tap to unfold the plain meaning

The floor plan: ~130 SM cook-stations share a ~50 MB L2, and all of them read from the ~94 GB HBM pantry at ~3.9 TB/s. Each station's own cutting board (SRAM) is tiny but blistering fast.

A GPU is three things: ~130 SM cook-stations, a Tensor Core appliance in each, and a memory hierarchy from tiny SRAM up to the big HBM pantry.

02

The SM: where kernels run

TL;DR · An SM (Streaming Multiprocessor) is one cook station. It runs threads in groups of 32 called warps, in lockstep, and hides memory waits by swapping to another warp.

"The GPU runs your code" really means: your kernels are chopped up and scheduled across ~130 SMs. Understanding one SM is understanding how work actually lands on the chip.

~130 cook stations

An H100 NVL has about 130 Streaming Multiprocessors (SMs). Each SM holds general-purpose CUDA cores (the everyday tools) plus one Tensor Core. Your kernels are scheduled across these SMs — fill them and the chip is busy; starve them and it idles.

Warps: 32 threads in lockstep

Each SM runs threads in groups of 32 called warps, in lockstep — all 32 execute the same instruction at once. This is the real unit of work on a GPU, not a single thread.

Hiding the wait

The GPU hides memory latency by swapping in another warp whenever one is waiting on data from memory. So while one warp stalls reading from HBM, the SM keeps busy on a different warp. Keeping enough warps in flight is how you keep the SM full.

Keywords — tap to unfold the plain meaning

Analogy ❝A cook station works on 32 identical plates at once — same chop, same motion, all in lockstep; that's a warp. When one batch of plates is stuck waiting on the pantry runner, the cook doesn't stand idle — they pull a different batch onto the board and keep moving. That swap is how the station stays busy even though the pantry is far away.

An SM runs warps of 32 threads in lockstep and hides memory latency by switching to another waiting warp. Keep the SMs full and the GPU earns its keep.

03

Tensor Cores: why GPUs are fast at LLMs

TL;DR · Each SM has one Tensor Core — a specialized appliance built for matrix multiplies. On Hopper they run FP8 via the Transformer Engine, which is why FP8 is so fast on your hardware.

An LLM is, underneath, a tower of matrix multiplies. The Tensor Core is the appliance built to do exactly that — and nothing makes a GPU faster at LLMs than feeding it well.

The matrix-multiply appliance

Alongside the general CUDA cores, each SM has one Tensor Core: a unit specialized for the matrix multiplies that dominate an LLM's forward pass. CUDA cores are the chef's knife; the Tensor Core is the stand mixer built for one job and brutally fast at it.

FP8 on Hopper

On Hopper, Tensor Cores run FP8 via the Transformer Engine — which is exactly why FP8 is so fast on your hardware. The narrower the number format the appliance natively supports, the more matrix-multiply throughput you get per second.

Keywords — tap to unfold the plain meaning

Why it matters When an earlier lesson said FP8 quantization "runs fast," this is the reason: the Transformer Engine gives Hopper's Tensor Cores native FP8 matrix-multiply support. The format isn't just smaller in memory — the appliance is wired to chew it at full speed.

One Tensor Core per SM does the matrix multiplies an LLM is made of. On Hopper it runs FP8 through the Transformer Engine — the hardware reason FP8 is fast.

04

The memory hierarchy: capacity vs bandwidth

TL;DR · Memory comes in tiers — tiny-but-instant SRAM on each SM (~19 TB/s) → shared L2 (~50 MB) → big-but-far HBM (~94 GB, ~3.9 TB/s). Small and close is fast; big and far is slow.

There is no free fast-and-big memory. The whole game of inference performance is keeping the data you're reusing as close to the cutting board as possible.

SRAM — the cutting board

On each SM sits SRAM (registers and shared memory): tiny, but instant — roughly ~19 TB/s. It's the cutting board right under the cook's hands. Data living here is essentially free to reuse.

L2 — the shared shelf

Between the stations and the pantry is a shared L2 cache of about ~50 MB. All SMs share it. It's a middle shelf — bigger than the cutting board, slower, but far closer than the pantry.

HBM — the pantry

HBM is the pantry: about ~94 GB of HBM3 at ~3.9 TB/s. Huge — it's where the weights and KV cache live — but a long walk away. Every trip you avoid is time saved.

Keywords — tap to unfold the plain meaning

The hierarchy, decoded — fast & small → big & slow

SRAM ~19 TB/s >> L2 ~50 MB > HBM ~94 GB @ ~3.9 TB/s

SRAMon-SM registers / shared memory; ~19 TB/s — about 5× HBM's bandwidth, but holds only kilobytes
L2~50 MB shared by all SMs; a middle tier between the cutting board and the pantry
HBM~94 GB HBM3 at ~3.9 TB/s; vast capacity, but the farthest and slowest to reach

Read it as a ladder: the closer to the SM, the faster but smaller. SRAM is ~19 TB/s but tiny; HBM is ~94 GB but only ~3.9 TB/s. Keeping reused data in SRAM instead of re-fetching from HBM is the core trick.

Analogy ❝The cutting board (SRAM) is right under the cook's hands — instant, but it holds a pinch. The shared shelf (L2) is a step away. The pantry (HBM) is enormous but down the hall: every ingredient you have to fetch from there is a trip that stalls the station. A great cook keeps what they're actively using on the board and only walks to the pantry when they truly must.

Registers/SRAM (~19 TB/s) → L2 (~50 MB) → HBM (~94 GB, ~3.9 TB/s). Fast means small and close; big means far and slow. Avoid pantry trips.

05

Tie it back: the whole course in one chip

TL;DR · Three things you already learned are now obvious: decode is memory-bound (waiting on HBM); fusion / FlashAttention keep data in SRAM; large batches keep the SMs busy.

This is the payoff. Every big optimization in the course was really a statement about this floor plan all along — you just hadn't seen the building yet.

Decode is memory-bound = waiting on HBM

In decode, you do little math per byte read, so the SMs sit idle waiting on the HBM pantry. That's what "memory-bound" means physically: the appliance is starved because ingredients arrive slowly from far away.

Fusion / FlashAttention keep data in SRAM

Kernel fusion and FlashAttention win by keeping intermediate data on the cutting board (SRAM) instead of round-tripping it to HBM. Fewer pantry trips, less stalling.

Batching keeps the SMs busy

Large batches give each SM more independent work — more warps in flight — so the cook stations stay full and the Tensor Cores stay fed. That's why throughput climbs with batch size.

Keywords — tap to unfold the plain meaning

The ridge, decoded

ridge point ≈ 214 FLOP/byte = peak computepeak HBM bandwidth

FLOP/bytearithmetic intensity: how much math you do for each byte you read from HBM
~214the ridge on your hardware — below it you're memory-bound, above it compute-bound
decodedoes far fewer than 214 FLOP/byte, so it lands left of the ridge → HBM-bound

The roofline ridge sits at ~214 FLOP/byte on your hardware. Work below that ratio is throttled by HBM bandwidth (decode); work above it is throttled by the Tensor Cores. The whole course is moving work past the ridge.

Decode waits on HBM; fusion/FlashAttention stay in SRAM; batching keeps SMs busy. The ~214 FLOP/byte ridge says which side of memory-bound you're on.

06

On YOUR cluster: the H100 (Hopper), concretely

TL;DR · Your 4×H100 box is four of these kitchens. Each H100 NVL: ~130 SMs, FP8 Tensor Cores, ~50 MB L2, ~94 GB HBM3 at ~3.9 TB/s, ridge ≈ 214 FLOP/byte.

These aren't abstract specs — they're the floor plan of the exact box you serve on. Memorize this one card and the rest of Part V has something concrete to hang on.

One H100 NVL, by the numbers

Per GPU: about 130 SMs; Tensor Cores with FP8 support via the Transformer Engine; roughly ~50 MB L2 cache; and ~94 GB of HBM3 running at ~3.9 TB/s. SRAM on each SM reaches ~19 TB/s. The roofline ridge lands near ~214 FLOP/byte.

4×H100 cluster Your serving box is four H100 NVL GPUs. That's ~4×130 = ~520 SMs and ~4×94 ≈ ~376 GB of HBM3 across the node, each GPU streaming at ~3.9 TB/s. The memory hierarchy — registers/SRAM (~19 TB/s) → L2 (~50 MB) → HBM (~94 GB, ~3.9 TB/s) — repeats once per GPU, and NVLink stitches them together for model parallelism.

Inspect one GPU on the 4×H100 box

nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv
# name           memory.total   memory.used
# NVIDIA H100 NVL  95830 MiB      1234 MiB   ← ~94 GB HBM3 per GPU

Keywords — tap to unfold the plain meaning

Check yourself — teach it back (recall, don't peek)

Explain to a colleague: "the whole course in one chip picture." What are the three parts of the GPU — and which one is each course optimization really about?
What is a warp, and how does an SM use warps to hide memory latency?
Order the memory tiers fast→big and give the headline number for each (SRAM, L2, HBM). Where do the weights and KV cache live?
Why is decode memory-bound, and why does it sit left of the ~214 FLOP/byte ridge?
Why is FP8 so fast on Hopper specifically?

Your box is 4× H100 NVL: ~130 SMs each, FP8 Tensor Cores, ~50 MB L2, ~94 GB HBM3 at ~3.9 TB/s, ridge ≈ 214 FLOP/byte. The whole course lives on this floor plan.