CUDA Kernels & Fusion — Inference Engineering

01

A forward pass is a sequence of kernels

TL;DR · One forward pass isn't a single GPU command — it's a long chain of kernels, each a tiny program the GPU launches across thousands of threads.

Last lesson the engine was the unit of thought. Drop one level and the engine dissolves into hundreds of individual GPU operations. Knowing what each one costs is where the next round of speed comes from.

A kernel is one operation

A CUDA kernel is one operation the GPU runs across thousands of threads: one trip to do one thing. A matrix multiply, an add, a softmax, an activation — each is its own kernel.

The pass is a long chain of them

A single forward pass fires off that whole chain in order. An activation here just means the intermediate vector a layer outputs and hands to the next kernel.

Each launch has a fixed tax

Beyond the math, every kernel carries a fixed launch overhead (~10–50 µs) just to start. Fire thousands of tiny kernels and that overhead alone starts to dominate the clock.

Keywords — tap to unfold the plain meaning

Analogy ❝Think of the GPU as a line cook and a kernel as one trip to fetch and do a single thing. The pantry is across town — slow to reach — and the cutting board is right at hand. Every kernel is a separate errand: walk out, do one task, walk back. A forward pass is a whole shift of these errands, run back to back.

A forward pass is a sequence of kernels — matmul, add, softmax, activation — and each kernel also pays a fixed ~10–50 µs launch tax just to start.

02

The cost: HBM round-trips on every kernel

TL;DR · Each kernel reads its inputs from HBM and writes its output back to HBM — so a chain of small kernels is a chain of slow round-trips across town.

Decode is memory-bound: the GPU spends its time waiting on memory, not on math. So the thing that matters isn't how clever each kernel is — it's how many times you cross the slow road to HBM.

Read from HBM, write to HBM

Each kernel reads its inputs from HBM and writes its output back to HBM. Read → compute → write, then the next kernel reads that result straight back out again. HBM is the GPU's main memory: the pantry across town, ~3.4 TB/s.

Two trips per kernel

Unfused, three kernels mean 3 kernels, 6 HBM trips — one read and one write apiece. The intermediate result of kernel one gets written all the way back to HBM only for kernel two to fetch it again.

On-chip memory is far faster

Right next to the compute sit SRAM and registers — the cutting board at hand, ~19 TB/s. If a value never had to leave the chip, you'd skip the whole crawl to and from HBM.

Three kernels, six trips. The result of each kernel is written all the way back to slow HBM only for the next kernel to read it back in.

Keywords — tap to unfold the plain meaning

Numbers, decoded

SRAM ~19 TB/s ÷ HBM ~3.4 TB/s ≈ 5.6× faster on-chip

3.4 TB/sHBM bandwidth — how fast the GPU can move data to its main memory across town
19 TB/son-chip SRAM/register bandwidth — the cutting board right beside the compute
≈ 5.6×keeping a value on-chip moves it roughly five-plus times faster than touching HBM

Because decode is memory-bound, the count of HBM round-trips is the real cost. Every trip you can keep on-chip instead of HBM is roughly a 5× speedup on that step's data movement.

Each kernel reads inputs from HBM and writes outputs back to HBM. Three small kernels = six round-trips. For memory-bound decode, that traffic is the cost.

03

Fusion — do more per trip

TL;DR · Kernel fusion merges several ops into one kernel, keeping the intermediates in fast on-chip SRAM — so three kernels and six HBM trips collapse to one kernel and two.

If the slow part is crossing the road to HBM, the fix isn't a faster road — it's fewer crossings. Fusion is exactly that: do several steps before you write anything back.

Merge ops into one kernel

Kernel fusion merges several ops into one kernel, so intermediate results live in on-chip SRAM/registers (~19 TB/s, vs HBM's ~3.4 TB/s) instead of being written out and read back.

One kernel, two trips

The fused version is 1 kernel, 2 HBM trips: read the inputs once, do all the work on-chip, write the final result once. The intermediates never touch HBM at all.

SwiGLU FFN is a classic

A SwiGLU FFN does its gate, activation, and elementwise multiply in a single kernel instead of three — a textbook fusion that turns three HBM round-trips into one.

Read inputs once, run gate → activation → multiply entirely on the chip, write the result once. The intermediates never visit HBM.

Keywords — tap to unfold the plain meaning

Analogy ❝Unfused, the cook makes three trips: fetch, season, return; fetch, cook, return; and so on — crossing town for every step. Fused, the cook fetches once and does all three on the board in one trip, then returns. Same dish, a third of the walking. The work didn't shrink; the round-trips did.

Fusion merges ops into one kernel so intermediates stay in SRAM, not HBM. SwiGLU's gate + activation + multiply fuse from 3 kernels / 6 trips down to 1 kernel / 2 trips.

04

FlashAttention — the famous fusion

TL;DR · FlashAttention fuses the whole attention into one kernel that streams over tiles in SRAM — never writing the giant n×n score matrix to HBM. Same result, a fraction of the memory traffic.

Attention is where the naive cost explodes, because the obvious way builds a full n×n matrix in HBM. FlashAttention is fusion applied to that exact problem — the single most important kernel optimization in modern serving.

The naive way is huge

The naive approach builds the full n×n score matrix in HBM — enormous for long contexts, with n² written to HBM. That write alone dominates the memory traffic of attention.

FlashAttention streams over tiles

FlashAttention fuses the whole attention into one kernel that streams over tiles in SRAM, one tile at a time (~19 TB/s), never writing that n×n matrix to HBM. Same result, a fraction of the traffic.

Tile through fast, skip the giant write

The whole trick: tile through fast SRAM, skip the giant HBM write. FlashAttention is fusion applied to attention — exact, not approximate, just memory-smart.

Left: the naive kernel materializes the whole n×n score matrix in HBM. Right: FlashAttention streams tiles through SRAM and writes only the final output — same answer, far less traffic.

Keywords — tap to unfold the plain meaning

Math, decoded

naive HBM write ∝ n² → FlashAttention HBM write ∝ n

nthe context length — how many tokens attention has to score against each other
n²naive attention materializes a full n-by-n score matrix and writes all of it to HBM
tilessmall blocks of the matrix processed one at a time, kept on-chip in SRAM
∝ nFlashAttention only writes the final output, so HBM traffic scales linearly, not quadratically

The naive kernel's HBM write grows with n² because it stores the entire score matrix. FlashAttention computes the same result tile-by-tile and never writes that matrix — so the giant quadratic write disappears.

FlashAttention is fusion applied to attention: tile through fast SRAM, skip the giant HBM write. Same exact result, a fraction of the memory traffic.

05

On your cluster

TL;DR · vLLM already ships fused kernels + FlashAttention, plus FP8 tensor-core kernels on your Hopper H100s — which is why decode sustains real throughput instead of drowning in launch overhead.

You don't write these kernels yourself — the engine ships them. But knowing they're there tells you which levers actually move the needle, and which numbers to expect when you measure.

The engine ships the kernels

vLLM ships fused kernels + FlashAttention and, on your Hopper H100s, FP8 tensor-core kernels. You get the fusion wins for free, baked into the runtime.

It's why decode keeps up

With fusion and FlashAttention doing the heavy lifting, your Qwen sustains ~108 decode tok/s rather than dying on launch overhead and HBM traffic.

Your lever: fewer bytes

The kernels are fixed, but you control how much data moves: fewer bytes via quantization (FP8/FP16) and GQA. Less data per trip is less HBM traffic on a memory-bound workload.

4×H100 note On the 4×H100 box, vLLM serves your Qwen with fused kernels, FlashAttention, and FP8 tensor-core kernels on Hopper. Measured: ~108 decode tok/s sustained — the fused path is what keeps decode off the launch-overhead-and-HBM-traffic floor. Your tunable lever is fewer bytes per trip: quantization (FP8/FP16) and GQA.

Keywords — tap to unfold the plain meaning

Check yourself

Finish the sentence for a colleague: "Fusing kernels speeds up decode because…"
Why does a chain of small unfused kernels hurt on memory-bound decode? (Hint: count the HBM trips.)
What does FlashAttention avoid writing to HBM, and where does the work happen instead?

Fusing kernels speeds up decode because it cuts HBM round-trips and per-kernel launch overhead: several ops run as one kernel with intermediates kept in fast on-chip SRAM. Since decode is memory-bound, fewer trips to slow HBM is a direct win.