Quantization Algorithms — Inference Engineering

01

Why naive rounding fails

TL;DR · The lazy way — round every weight to the nearest low-bit value — breaks because a few outlier weights round badly and their error spreads. At INT4, that can wreck the model.

Last lesson gave you the number formats. Now the question is how you actually move weights into them. The obvious move — just round — is exactly the one that fails, and seeing why sets up every algorithm that follows.

The naive baseline: RTN

RTN (Round-To-Nearest) is the simplest possible scheme: take each weight and round it to the nearest value the low-bit format can represent. No calibration, no cleverness — just rounding. It's the baseline everything else is measured against.

Outliers ruin it

Most weights round fine. The trouble is a few outlier weights with unusually large magnitudes. Rounding those badly introduces error, and because layers feed into layers, that error propagates forward through the network. At INT4 especially, the buckets are so coarse that naive RTN can wreck a model outright.

So the fix is never "round harder"

Every algorithm in this lesson — GPTQ, AWQ, SmoothQuant — is a different answer to the same problem: how do you keep those few troublesome weights and activations from poisoning the result when you round everything to low bits?

Keywords — tap to unfold the plain meaning

Analogy ❝Picture a line cook reading a recipe and rounding every quantity to the nearest pinch. For flour and water, rounding barely matters — the dish survives. But round the saffron or the chili the same careless way and the whole plate is off. A few ingredients carry the dish; round those wrong and the error follows the food all the way to the table.

RTN just rounds. It fails because a few outlier weights round badly and their error propagates — worst at INT4. The rest of the lesson is three ways to fix that.

02

GPTQ — compensate the error

TL;DR · GPTQ quantizes one layer at a time and, after rounding each weight, nudges the remaining weights to cancel the error it just made — using Hessian info to know which weights matter.

If rounding one weight introduces error, the trick is to make the next weights absorb it. GPTQ doesn't fight the rounding — it pays the debt forward.

Quantize, then compensate

GPTQ works layer by layer. It rounds one weight, measures the error that created, and adjusts the still-unquantized weights in that layer to cancel it out. Each rounding is compensated by the next adjustment, so error doesn't pile up.

It uses Hessian information

To know which weights to lean on, GPTQ uses Hessian information — an estimate of how sensitive the layer's output is to each weight. Sensitive weights get handled with more care; forgiving ones absorb the slack.

It needs calibration data

That sensitivity estimate isn't free: GPTQ requires calibration data — a small sample of real inputs run through the layer so it can see which weights actually matter. It's a weight-only method, typically used to reach INT4.

Keywords — tap to unfold the plain meaning

Analogy ❝After rounding the flour down a touch, the line cook tastes the batter and adds a little more butter to keep the recipe balanced. Each rounding error is quietly compensated by the next adjustment, so the finished dish lands right even though no single quantity was measured perfectly.

GPTQ rounds one weight at a time and nudges the rest to cancel each rounding error, guided by Hessian sensitivity. Weight-only, needs calibration data, reaches INT4.

03

AWQ — protect the salient few

TL;DR · AWQ notices that roughly 1% of weights — the ones multiplying the largest activations — carry most of the quality. It scales those to survive quantization intact while keeping the rest low-bit.

GPTQ asks "how do I undo rounding error?" AWQ asks a sharper question: "which weights should I never let round badly in the first place?" The answer turns out to be a tiny, identifiable few.

Not all weights are equal

AWQ (Activation-aware Weight Quantization) is built on one observation: a small set of salient weights — about 1% — matters far more than the rest. These are the weights that multiply the largest activations, so errors in them have outsized impact on the output.

Scale them to survive

Rather than store that 1% in higher precision, AWQ scales the salient weights up before quantizing so they keep their detail through rounding, then scales back afterward. Everything else stays plain low-bit. It's weight-only, like GPTQ, and aimed at INT4.

"Activation-aware" is the key word

The cleverness is using the activations — the intermediate vector a layer outputs — to decide which weights are salient. A weight is important not on its own, but because of how big the thing it multiplies tends to be.

Keywords — tap to unfold the plain meaning

Analogy ❝Measure the saffron and the chili to the milligram; eyeball the potatoes. The line cook protects only the few ingredients the dish actually hinges on, and is gloriously sloppy with everything else — because that's exactly where sloppiness is free.

AWQ finds the ~1% of salient weights (those multiplying the biggest activations) and scales them to survive quantization. Weight-only, activation-aware, reaches INT4.

04

SmoothQuant — move the outliers

TL;DR · Activations are harder to quantize than weights because their outliers are wild. SmoothQuant rescales per channel to migrate that difficulty out of the activations and into the weights — so both quantize easily, enabling W8A8.

GPTQ and AWQ only touch weights. But if you want to quantize the activations too — and you do, for speed — you hit a new wall: activation outliers are brutal. SmoothQuant's move is to relocate the problem.

Activations have the wild outliers

An activation is the intermediate vector a layer outputs. Weights tend to be well-behaved; activations are where the truly large outliers live, which makes them the hard part to quantize.

Rescale per channel to share the pain

SmoothQuant rescales per channel — multiplying activations down and the matching weights up by the same factor, which leaves the math unchanged. This migrates the outlier difficulty out of the activations and into the weights, so neither side has a wild range any more and both quantize cleanly.

This unlocks W8A8

Because both sides are now tame, SmoothQuant enables W8A8 — 8-bit weights and 8-bit activations. That's the regime that matters on hardware without native FP8 support, where you still want 8-bit activations for throughput.

Multiply the activation channel down by a factor and the weight channel up by the same factor: the product is identical, but the outlier has moved into the tamer weights. Now both sides fit cleanly in 8 bits.

Keywords — tap to unfold the plain meaning

Math, decoded

Y = (X · diag(s)^-1) · (diag(s) · W) = X · W

Xthe activations going into the layer — where the wild outliers live
Wthe layer's weights — naturally tame and easy to quantize
sa per-channel scale factor, chosen so the activation outliers shrink
X · diag(s)^-1activations divided by s per channel → outliers tamed, easy to quantize
diag(s) · Wweights multiplied by s per channel → they absorb the range instead
= X · Wthe two scale factors cancel, so the layer's output Y is exactly unchanged

Divide activations and multiply weights by the same per-channel factor s. The output is mathematically identical, but the hard-to-quantize range has been migrated from activations into weights — so both now fit 8 bits, giving W8A8.

Analogy ❝One ingredient comes wildly over-concentrated — a chili paste ten times too strong to measure by eye. Instead of struggling with it, the line cook dilutes the paste and stiffens the dough by the same factor. The finished dish tastes exactly the same, but now every quantity is back in an easy-to-eyeball range. The difficulty didn't vanish — it just moved somewhere the cook can handle it.

Activations hold the wild outliers. SmoothQuant rescales per channel to migrate that difficulty into the tamer weights, leaving the math unchanged — which makes W8A8 possible.

05

Three fixes, side by side

TL;DR · Same enemy — outliers — three different escapes: GPTQ cancels the error, AWQ protects the salient few, SmoothQuant moves the difficulty out of activations into weights.

It's easy to mix these up. The clean way to hold them apart is by what each one touches and what trick it plays — then the right choice almost picks itself.

Two are weight-only, one does both

GPTQ and AWQ quantize weights only and aim at INT4. SmoothQuant is the one that also quantizes activations, which is why it's the path to W8A8.

Each plays a different trick

GPTQ compensates rounding error using Hessian sensitivity. AWQ protects the ~1% salient weights by scaling them. SmoothQuant migrates outlier difficulty from activations into weights via a per-channel rescale.

RTN is the baseline that fails. GPTQ and AWQ are weight-only INT4 methods with different tricks; SmoothQuant is the weights-and-activations method that reaches W8A8.

Keywords — tap to unfold the plain meaning

GPTQ compensates, AWQ protects, SmoothQuant migrates. GPTQ and AWQ are weight-only INT4; SmoothQuant does weights and activations for W8A8.

06

Choosing on your cluster

TL;DR · Your Qwen runs FP8 (W8A8), whose wide dynamic range tolerates outliers without these tricks. They matter only when you go lower or run non-FP8 hardware.

The point of all this isn't to memorize three papers — it's to pick a quantized checkpoint on purpose. So when does each one actually earn its place on a real box?

Why your default needs none of them

On the cluster, your Qwen runs FP8 (W8A8). FP8's wide dynamic range simply tolerates the outliers that wreck INT formats — so it needs none of GPTQ, AWQ, or SmoothQuant. They earn their keep only when you push lower than FP8 or land on hardware without it.

When each one wins

Reach for AWQ or GPTQ at INT4 when you need to fit a bigger model on a smaller GPU — INT4 roughly halves the memory of 8-bit. Reach for SmoothQuant at INT8 when you're on non-FP8 hardware but still want 8-bit activations for throughput.

Cluster note On the 4×H100 box, the served checkpoint is Qwen3.6-27B-FP8 — already W8A8 in FP8, so it skips these algorithms entirely; FP8's dynamic range absorbs the outliers. You'd only switch to AWQ/GPTQ-INT4 to squeeze a larger model onto fewer GPUs, or to SmoothQuant-INT8 if you redeployed to a GPU generation without native FP8.

Run it yourself

The runnable companion is the day14 notebook, quantization-gptq-awq-smoothquant, which walks through all three on real weights so you can watch RTN fail and each fix recover quality.

References — the three papers

GPTQ         Frantar et al.   arXiv:2210.17323
AWQ          Lin et al.       arXiv:2306.00978
SmoothQuant  Xiao et al.      arXiv:2211.10438

Keywords — tap to unfold the plain meaning

Check yourself

Explain to a colleague: "We can't just round weights to 4 bits because…" — finish the sentence.
GPTQ, AWQ, SmoothQuant — match each to its one-line trick (compensate / protect / migrate).
Why does your FP8 Qwen need none of these, and what would make you switch to AWQ-INT4 or SmoothQuant-INT8?

A few outlier weights round badly and wreck quality — so we use GPTQ (compensate via Hessian), AWQ (protect ~1% salient weights), or SmoothQuant (shift activation outliers into weights). FP8 sidesteps all three.