Each step below: commit a guess, then hit Reveal. Predicting first (even a wrong guess) is what makes it stick.
Today's win: you'll predict a GPU's internals (SMs, Tensor Cores, the memory
hierarchy) and connect each to a lesson you've already done.
The setup
We've leaned on "the pantry" (HBM) and "the cook" (compute) all course. Time to open the box: what's
actually inside the GPU?
Step 1: the compute units
Step 2: the LLM workhorse
Recall (cover the screen): the GPU memory hierarchy, fast→big. Registers / SRAM (tiny, on each SM, ~19 TB/s) → L2 (~50 MB, shared) → HBM (~94 GB, ~3.9 TB/s). Always capacity vs bandwidth. Decode waits on HBM; fusion/FlashAttention keep data in SRAM.(tap/hover to check)
Step 3: the memory tiers
Step 4: tie it back
Step 5: your hardware
On YOUR cluster your hardware
Each GPU is an H100 NVL (Hopper): ~130 SMs with FP8 Tensor Cores + Transformer
Engine, ~50 MB L2, and ~94 GB HBM3 @ ~3.9 TB/s. Those exact numbers set your roofline ridge (~214), KV
capacity, and why FP8 + FlashAttention win here. Your Lab →
Explain to a colleague: "The whole course in one chip picture…" …a GPU has ~130 SMs (compute, incl. FP8 Tensor Cores) fed by a memory hierarchy: tiny fast SRAM on each SM, then L2, then big slow HBM. Decode is memory-bound = waiting on HBM; fusion/FlashAttention keep data in SRAM; batching keeps the SMs busy. The hardware explains every optimization.(tap/hover)
I'm your teacher: ask me anything. Want to map a specific optimization to the exact part of the chip it targets?
The hardware floor, where every optimization in this course physically lives.
Today's win: you'll explain a GPU's internals (the SMs as compute units, Tensor Cores,
and the memory hierarchy registers → SRAM → L2 → HBM) so memory-bound decode, fusion, and
FlashAttention all have a concrete physical home.
The picture: the kitchen building itself
We've talked about the pantry and the van for the whole course. Now look
at the building. A GPU is a kitchen with ~130 cook stations (SMs), each holding general tools
(CUDA cores) and one specialized appliance (a Tensor Core for matrix multiplies). The pantry is
HBM; each station's cutting board is tiny, instant SRAM.
a cook station
SM (Streaming Multiprocessor): runs your kernels
the matrix-multiply appliance
Tensor Core, the LLM workhorse
the pantry across town
HBM (~94 GB, ~3.9 TB/s on your H100 NVL)
the cutting board at the station
SRAM / registers (tiny, ~19 TB/s)
1 · The SM: where kernels run
A GPU is an array of Streaming Multiprocessors (an H100 has ~130). Each SM runs threads
in groups of 32 called warps, in lockstep. Your kernels
are scheduled across these SMs, and the GPU hides memory latency by swapping in another warp whenever one is
waiting on data.1
Many SMs, each with its own cores + Tensor Core + scratchpad, all pulling from one big shared
HBM. Keeping those SMs fed is the whole performance game.
2 · Tensor Cores: why GPUs are fast at LLMs
The Tensor Core is a dedicated unit that does small matrix multiplies in one shot. LLMs
are almost entirely matrix multiplies, so Tensor Cores are where the FLOPs come from. On Hopper they
run FP8 (via the Transformer Engine), which is why
FP8 is so fast on your hardware.1
3 · The memory hierarchy: capacity vs bandwidth
Memory comes in tiers: tiny+instant registers and SRAM (~19 TB/s) on
each SM, a shared L2 (~50 MB), and big+slower HBM (~94 GB, ~3.9 TB/s). The
trade is always capacity vs bandwidth.2 This single picture
explains the whole course:
Big batches keep the SMs (compute) busy instead of idle.
The pyramid behind everything: data is fast when it's small and on-chip, slow when it's big and
in HBM. Every optimization is a move up this pyramid.
On YOUR cluster: the H100 (Hopper), concretely your hardware
Each of your 4 GPUs is an H100 NVL (Hopper): ~130 SMs with FP8 Tensor
Cores + Transformer Engine, ~50 MB L2, and ~94 GB HBM3 at ~3.9 TB/s. Those exact numbers are
what set your roofline ridge (~214 FLOP/byte), your
KV-cache capacity, and why FP8 + FlashAttention are
such wins here. The hardware is the constraint every lesson has been dancing around. · Your Lab →