Quantization: Number Formats — Inference Engineering

What quantization actually is

TL;DR · A model is billions of numbers — store each in fewer bits and you round every value onto a coarser grid. Halve the bits, halve the footprint.

Quantization isn't a trick you bolt on; it's just choosing how many digits to keep per number. The whole lesson is what you gain by keeping fewer, and what breaks if you keep too few.

Fewer bits per number

A model is billions of numbers. Quantization stores each one in fewer bits — FP16 at 2 bytes → FP8 at 1 byte → INT4 at ½ byte — and rounds each value onto a coarser grid. Halve the bits, halve the footprint, and snap each number to a coarser grid.

Coarser grid, thinner artifact

FP16 might record a weight as 237.4; FP8 records it as ≈ 240. Fewer distinct values are representable, so each number lands on the nearest rung of a coarser ladder. The binary on disk and in memory gets proportionally smaller.

Keywords — tap to unfold the plain meaning

Each format keeps fewer bits per weight, so the on-disk and in-memory artifact shrinks proportionally — at the cost of a coarser representable grid.

Analogy ❝Picture the recipe binders in a faraway pantry. FP16 writes every measurement as "237.4 g"; FP8 just writes "≈ 240 g". Fewer digits per measurement, so the binder gets thinner — and the van that hauls binders out to the line cook carries less paper per trip.

Quantization stores each weight in fewer bits and snaps it to a coarser grid. FP16 = 2 bytes, FP8 = 1 byte, INT4 = ½ byte. Halve the bits, halve the footprint.

Two levers: weights and the KV cache

TL;DR · Fewer-bit weights mean fewer bytes hauled per token → faster decode. A fewer-bit KV cache means more sequences fit → more concurrency.

Quantization pays off in two separate places, and they're worth keeping apart in your head: it makes each token faster to produce, and it lets you serve more of them at once.

Lever 1 — quantize the weights → faster decode

Decode is memory-bound: its cost is the bytes of weights hauled per token. FP8 weights are half the bytes, so you get roughly 2× decode throughput — and the model fits on fewer GPUs.

Lever 2 — quantize the KV cache → more concurrency

An FP8 KV cache halves the bytes-per-token factor of the cache → roughly half the cache size → more sequences, or longer context, on the same GPU. This is concurrency, not raw speed.

Keywords — tap to unfold the plain meaning

Analogy ❝Back to the pantry. Thinner recipe binders (the weights) mean the van hauls less paper per trip out to the line cook — every order gets plated faster, because the bottleneck was always the hauling, not the cooking. Thinner index cards (the KV cache) mean more of them fit on the shelf — so the cook can keep many more orders in flight at once.

Two levers. Smaller weights → fewer bytes per token → ~2× faster decode. Smaller KV cache → half the size → more concurrent sequences. Decode is memory-bound, so bytes are the currency.

The catch: precision and outliers

TL;DR · The typical values round fine. The danger is a handful of outliers — rare huge values whose rounding error blows up the result if you treat them like everything else.

If every weight were ordinary, quantization would be free. It isn't free because of a small minority of values that refuse to fit on the coarse grid without wrecking the output.

It's not the typical values

The difficulty isn't the millions of typical values, which round fine onto the coarser grid. It's a handful of outliers: a few unusually large values whose rounding error blows up the rest of the computation.

Keep the saffron exact

The fix is to give outliers special treatment instead of crushing them onto the same grid. Common methods: per-channel scales, SmoothQuant, AWQ, and simply keeping outliers in 16-bit while the rest goes low-precision.

Keywords — tap to unfold the plain meaning

Analogy ❝Most pantry measurements survive a coarse measuring cup just fine — a cup of flour rounded to the nearest cup is still bread. But a pinch of saffron needs to be exact: round it to the nearest tablespoon and the dish is ruined. The outliers are the saffron. You keep those few precise, and let everything else snap to the coarse cup.

Typical values round fine; outliers are the saffron — a few huge values whose rounding error blows up the result. Per-channel scales, SmoothQuant, AWQ, and 16-bit outliers keep them precise.

The format map: what to reach for

TL;DR · FP8 (E4M3) on H100-class GPUs is near-lossless and quantizes everything. INT4 weight-only is for memory-starved GPUs. The notation W<weights>A<activations> tells you what's quantized.

There isn't one quantization — there's a small menu, and the right pick depends on your GPU and what's actually starving you. Learn to read the WxAy notation and the menu reads itself.

Read the WxAy notation

W8A8 means 8-bit weights and 8-bit activations (the intermediate vector a layer outputs). W4A16 means 4-bit weights but 16-bit activations — weight-only quantization. The W number and the A number are the bit widths of each.

The four formats worth knowing

FP8 (E4M3), W8A8 — 8-bit, quantizes weights + activations + KV, ≈ lossless, for H100+ (your cluster). INT8 + SmoothQuant — 8-bit weights + activations, ~1–3% quality cost, broad hardware with integer tensor cores. INT4 (GPTQ/AWQ), W4A16 — 4-bit weights only, small loss if you protect salient weights, for memory-bound or smaller GPUs. FP4 / NVFP4 — 4-bit, frontier, needs more care, for newest Blackwell-class hardware.

Two rules of thumb

Reach for weight-only INT4 (W4A16) when you're memory- or bandwidth-starved at modest batch. Reach for W8A8 (FP8 / INT8) when you want faster compute under heavy continuous batching on big GPUs.

Keywords — tap to unfold the plain meaning

Pick rule Memory-starved at modest batch? Use W4A16 (INT4 weight-only) — you cut the bytes hauled per token without paying for activation quantization. Compute-bound under heavy continuous batching on big GPUs? Use W8A8 (FP8 or INT8) — you light up the faster low-precision tensor cores for prefill and decode.

WxAy = bit width of weights and activations. FP8/E4M3 W8A8 is near-lossless on H100+. INT4 W4A16 is for memory-starved GPUs at modest batch; W8A8 for compute-bound heavy batching.

On your cluster

TL;DR · Qwen3.6-27B-FP8 runs in E4M3: ~27 GiB of weights fit on one H100 with no tensor-parallel split, and an FP8 KV cache halves bytes-per-token to enable ~2× concurrency.

This is the lesson made concrete on the 4×H100 box: real byte counts, real TFLOPS, and one subtlety about head_dim that decides whether FP8 helps prefill or quietly hurts it.

FP8 weights fit on one H100

Qwen3.6-27B-FP8 stores its weights in FP8 (E4M3) at ~27 GiB, versus ~54 GiB in FP16. That fits on a single 94 GiB H100 with no tensor-parallel split needed — what used to require splitting across GPUs now runs on one.

FP8 KV cache → ~2× concurrency

With --kv-cache-dtype fp8, each KV element is 1 byte: 128 KiB/token versus 256 KiB in FP16. Half the cache per token enables roughly 2× concurrency on the same GPU.

FP8 tensor cores help prefill too

The H100's FP8 tensor cores hit 3,341 TFLOPS versus 1,671 TFLOPS in BF16 — about 2× the BF16 FLOPs, so prefill speeds up as well, not just decode.

4×H100 note Measured on the 4×H100 box. FP8 weights (E4M3): ~27 GiB vs ~54 GiB FP16 → fits on one 94 GiB H100, no TP split. KV: --kv-cache-dtype fp8 → 128 KiB/token vs 256 KiB → ~2× concurrency. Compute: FP8 cores 3,341 TFLOPS vs BF16 1,671 TFLOPS. Subtlety: this model's head_dim = 256 — FP8 KV clearly speeds decode, but prefill can run slightly behind BF16 there, so benchmark before you assume FP8 wins everywhere.

Launch with an FP8 KV cache

vllm serve Qwen3.6-27B-FP8 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768

Keywords — tap to unfold the plain meaning

Math, decoded

bytes/tok_FP16 = 256 KiB → bytes/tok_FP8 = 2562 = 128 KiB

bytes/tokhow many bytes of KV cache each generated token consumes
FP16 = 2 B/eltwo bytes per KV element gives 256 KiB per token for this model
FP8 = 1 B/elone byte per KV element halves it to 128 KiB per token
÷2half the bytes per token → roughly twice as many sequences fit → ~2× concurrency

Switching the KV cache from FP16 (2 bytes/element) to FP8 (1 byte/element) halves bytes-per-token from 256 KiB to 128 KiB — and since the cache is what caps how many sequences fit, that buys roughly 2× concurrency.

Check yourself

Picture the coarser grid and the saffron, then answer from memory: explain the outlier problem in quantization.
Finish the sentence to a colleague: "Quantization is basically free for us because…" (Hint: FP8 is ~lossless yet halves both the weights hauled per token — faster, memory-bound decode — and the KV cache — more concurrency — while modern methods keep the few outlier values precise, which is where quality would otherwise break.)
What does W4A16 mean, and when would you reach for it over W8A8?

On the 4×H100 box, FP8 E4M3 puts Qwen3.6-27B in ~27 GiB on one H100, halves the KV cache to 128 KiB/token for ~2× concurrency, and doubles tensor-core FLOPs — but check head_dim=256 before assuming FP8 wins prefill.

Reached the end — nice. This lesson now counts toward your progress.