Dropping bits without dropping quality, one guess at a time.
Each step below: commit a guess, then hit Reveal. Predicting first, even a wrong guess, is what makes it stick.
Today's win: you'll predict why naive rounding fails and how GPTQ, AWQ, and
SmoothQuant each fix it: enough to choose a quantized checkpoint on purpose.
The setup
Lesson 24 said quantization writes each weight with fewer bits. The naive way is just to round every
weight to the nearest low-bit value (RTN).
Step 1: why RTN struggles
Step 2: GPTQ's idea
Recall, cover the screen: what do all three algorithms have in common? They're all outlier strategies. Naive rounding ruins a few large/important values. GPTQ compensates the error, AWQ protects the salient weights, and SmoothQuant migrates activation (an activation is the intermediate vector a layer outputs) outliers into weights. Each uses a small calibration set to learn what matters.(tap/hover to check)
Step 3: AWQ's idea
Step 4: SmoothQuant's idea
Step 5: which applies to you?
On YOUR cluster context
Your Qwen is FP8 (W8A8): its wide dynamic range lets it tolerate outliers without these
tricks (FP8 "just works"). They matter when you go lower: AWQ/GPTQ-INT4 to fit a bigger model on a
smaller GPU, or SmoothQuant-INT8 on non-FP8 hardware. Now ...-AWQ in a checkpoint name means
something to you. Your Lab →
Explain to a colleague: "We can't just round weights to 4 bits because…" …a few outlier weights round badly and wreck quality. So we use GPTQ (compensate the rounding error using Hessian info), AWQ (protect the ~1% salient weights tied to big activations), or SmoothQuant (shift activation outliers into the weights so both quantize). Each uses calibration data to learn what to protect.(tap/hover)
I'm your teacher. Ask me anything. Want to compare FP8 vs AWQ-INT4 of your model on quality + memory?
GPTQ, AWQ, SmoothQuant: how to drop bits without dropping quality.
Today's win: you'll explain why naive rounding loses accuracy, and how the three big
post-training methods (GPTQ, AWQ, SmoothQuant) each beat it by handling outliers, so you can pick a
quantized checkpoint with confidence.
The picture: which measurements must stay precise?
Lesson 24
said quantization = writing each number with fewer significant figures. But you learned the catch: most
measurements survive a coarse cup, while a pinch of saffron (an outlier) ruins the dish if rounded.
These algorithms are the smart strategies for which values to protect and how to
compensate for the rounding you do.
round everything to the nearest cup
RTN (round-to-nearest): the naive baseline
adjust later steps to cancel the error
GPTQ
keep the saffron exact
AWQ (protect salient weights by scaling them to survive low-bit rounding)
move the hard-to-measure part elsewhere
SmoothQuant (migrate outliers)
1 · The baseline: round-to-nearest, and why it hurts
The naive method, RTN, just rounds every weight to the nearest value on the low-bit
grid. It's instant and free, but accuracy slips, because a few outlier weights round
badly and their error propagates. At INT4 especially, naive RTN can wreck a model.1
Naive RTN treats every weight equally. The few outliers, the saffron, are exactly what breaks.
Every good algorithm is an outlier strategy.
2 · GPTQ: compensate the error as you go
GPTQ quantizes one layer at a time and, after rounding each weight, nudges the
remaining weights to cancel the error it just introduced, using second-order, or Hessian, information
(an estimate of how sensitive the output is to each weight) about which adjustments matter. It needs a small calibration dataset to estimate that.
Result: solid INT4 weights with little quality loss.1
Pantry: after rounding the flour down, add a touch more butter to keep the
recipe balanced; each rounding is compensated by the next adjustment.
3 · AWQ: protect the weights that matter
AWQ (Activation-aware Weight Quantization) notices that the weights multiplying the
largest activations (an activation is the intermediate vector a layer outputs) matter most. It identifies that ~1% of salient weights and
scales them so they survive quantization intact, while the rest go low-bit.2
Pantry: measure the saffron and chili to the milligram; eyeball the
potatoes. Protect the few ingredients the dish actually hinges on.
4 · SmoothQuant: move the difficulty
Quantizing activations (not just weights) is hard because activations have wild outliers.
SmoothQuant rescales per channel to migrate that difficulty from the activations
into the weights, so both become easy to quantize. This enables 8-bit weights and activations
(W8A8).3
Three angles on the same enemy: outliers. GPTQ & AWQ make weight-only low-bit work;
SmoothQuant unlocks quantizing activations too.
On YOUR cluster: where this applies context
Your Qwen runs FP8 (W8A8), whose wide dynamic range often tolerates outliers
without these tricks, so FP8 largely "just works." These algorithms earn their keep when you go
lower: an AWQ- or GPTQ-INT4 checkpoint to fit a bigger model on a smaller GPU, or
SmoothQuant for INT8 on hardware without FP8. Knowing them lets you read a checkpoint's name
(e.g. ...-AWQ) and know exactly what you're getting. · Your Lab →
Study next: sources & lab companionAWQ, Lin et al. Runnable companion:
day14 notebook, with GPTQ, AWQ, SmoothQuant compared.
Check yourself (recall, don't peek)
I'm your teacher. Ask me anything. Want to compare an FP8 vs AWQ-INT4 checkpoint of
your model on quality + memory? Just ask.