Speculative Decoding — Inference Engineering

01

Why idle compute is the opening

TL;DR · Decode is memory-bound: each token streams the whole model from memory but does little math, so the GPU's compute sits mostly idle. Speculative decoding spends that idle compute.

Every other trick in this part fought the memory wall by moving less data. This one is different — it accepts the streaming cost, then squeezes free work out of the compute that was already going to waste.

Decode barely touches the math units

During decode, generating one token means the GPU reads every weight from memory but performs only a sliver of arithmetic with them. The bottleneck is memory bandwidth, not the math units — so those units sit mostly idle while the weights stream past.

Idle compute is an opportunity, not waste

If the weights are already being streamed for one token, the GPU has spare compute to score several positions in the same pass — almost for free. The whole idea of speculative decoding is to fill that idle compute with verification work.

Keywords — tap to unfold the plain meaning

Analogy ❝Decode is a line cook whose ingredients live in a faraway pantry. Each token, they walk the long aisle to fetch the whole model and walk back — the walk dominates, the actual chopping is trivial. Their knife hand is idle most of the trip. Speculative decoding says: since you're already making the walk, carry back enough to prep several bites at once. The walk costs the same; the idle knife finally does some work.

Decode is memory-bound, so the math units are mostly idle. Speculative decoding fills that idle compute with verification — several tokens per pass instead of one.

02

Draft, then verify

TL;DR · A cheap draft model guesses K tokens ahead; the target model verifies all K in a single forward pass and keeps the longest correct prefix.

The shape is simple: guess fast, check once, keep what survives. A slow model checking a fast model's homework — and getting several tokens for the price of one check.

The draft model proposes K tokens

A small, fast draft model runs ahead and proposes K candidate tokens — a short guessed continuation. Being small, it churns these out cheaply, far faster than the real model could.

The target model verifies all K at once

The real target model takes those K guesses and checks them in one verification pass — a single forward pass over all K positions at once. It then accepts every drafted token that matches its own next-token choice, stopping at the first mismatch.

Keep the longest correct prefix

The target accepts the longest run of guesses it agrees with — the accepted prefix — and resamples the one token where the draft first went wrong. One verify pass can therefore commit several tokens instead of just one.

Keywords — tap to unfold the plain meaning

The draft proposes four tokens; the target verifies all four in a single pass, accepts the longest matching prefix (A, B, C), and resamples where the draft first diverged (D).

Draft guesses K, target verifies all K in one pass, keeps the longest correct prefix. Several committed tokens for the cost of a single target pass.

03

Lossless, not approximate

TL;DR · Rejection sampling makes the accept/resample rule provably reproduce the target model's own next-token distribution — the output is mathematically identical, not a quality trade.

The natural worry: "won't a cheap draft degrade my answers?" No. The acceptance math is built so the final text is exactly what the target alone would have produced. Speed, for free.

Each token is accepted with a bias-correcting probability

A drafted token isn't accepted blindly. Under rejection sampling, each one is accepted with a probability that corrects the draft's bias — it compares the draft's probability for that token against the target's and accepts proportionally.

Rejected tokens are resampled from a corrected distribution

When a token is rejected, it's resampled from a corrected distribution (the target minus draft, clamped to be valid). This bookkeeping keeps the output provably identical to the target model's own next-token distribution.

So it's lossless

The result is lossless: mathematically identical to normal decoding, not an approximation. You change how fast tokens come out, never which tokens come out.

Keywords — tap to unfold the plain meaning

Math, decoded

accept with prob min(1, p(x)q(x)) → if rejected, resample from norm(p−q)⁺

q(x)the draft model's probability for the proposed token x
p(x)the target model's probability for that same token x
min(1, p/q)accept the draft's token with this chance — if the target likes it at least as much, always accept
(p−q)⁺the leftover where the target wanted more than the draft offered; clamp negatives to zero
norm(·)renormalize that leftover into a valid distribution, then sample the replacement token from it

Accepting at min(1, p/q) and resampling rejects from the normalized positive part of (p − q) is exactly engineered so the tokens you emit follow p, the target's distribution — provably identical to running the target alone.

Rejection sampling accepts each draft token with a bias-correcting probability and resamples rejects from a corrected distribution — so the output is provably identical to the target's. Lossless.

04

Three ways to draft

TL;DR · The guesses can come from a separate small model, from lightweight heads on the target itself (Medusa, EAGLE), or from n-gram lookup of repetitive text already in the context.

"Draft model" is the textbook framing, but in practice the draft can be almost anything cheap that's usually right. The cheaper and more aligned the guesser, the bigger the win.

A separate draft model

The classic recipe: a small sibling model of the target — same family, far fewer parameters. It runs ahead independently and proposes the K tokens, as in Leviathan et al.'s original speculative decoding.

Self-speculation: heads on the target

Self-speculation drops the separate model. Instead, lightweight extra prediction heads bolted onto the target predict several tokens ahead from its own hidden states — this is what Medusa and EAGLE do.

N-gram / prompt lookup

N-gram (or prompt lookup) needs no model at all: it proposes the next tokens by matching against repetitive text already in the context. Cheap and strong when the output echoes the input — code, JSON, edits.

Keywords — tap to unfold the plain meaning

Refs Leviathan et al., Fast Inference from Transformers via Speculative Decoding (arXiv:2211.17192) is the foundational paper. Medusa (arXiv:2401.10774) and EAGLE (arXiv:2401.15077) introduce the self-speculation heads that live on the target model itself.

Three draft sources: a separate small sibling model, self-speculation heads on the target (Medusa, EAGLE), or n-gram lookup of repetitive context. Anything cheap that's usually right.

05

Acceptance rate is everything

TL;DR · Speedup depends entirely on the acceptance rate — how often draft tokens match the target. High alignment (e.g. 82% in code) yields ~4 bonus tokens per verify; low alignment can make it a net slowdown.

This is the lever, and it's binary in spirit: when the draft guesses well, you fly; when it guesses badly, you've added work and gained nothing. Same with batching — if the GPU is already busy, there's no idle compute left to spend.

High acceptance, many bonus tokens

The acceptance rate is the fraction of drafted tokens the target accepts. On structured text like code it can hit 82%, yielding roughly 4 bonus tokens per verify pass — every committed token costs a fraction of a target pass.

Low acceptance, no win — or a loss

If the draft and target disagree often, the accepted prefix is short, the wasted draft and verify work piles up, and you get negligible gains or an outright slowdown.

Batching already eats the idle compute

Speculative decoding feeds on idle compute. Under high batch concurrency, batching already saturates the math units, so there's little idle compute left to spend — gains shrink toward break-even or worse.

Keywords — tap to unfold the plain meaning

Math, decoded

expected accepted ≈ 1 − α^K+11 − α (per verify pass)

αthe per-token acceptance rate — chance the target keeps a given draft token
Kdraft length: how many tokens the draft proposes before each verify
α^K+1acceptance fails geometrically the further you guess; long runs of agreement get rare fast
the ratioaverage number of tokens committed per verify pass — grows with α, capped near K

Tokens committed per pass rises sharply as α nears 1: at α = 82% you commit on the order of 4 per pass. As α drops, the ratio collapses toward 1, and the extra draft + verify overhead can make you slower than plain decode.

Cluster note Measured with vLLM + z-lab/Qwen3.6-27B-DFlash (a block-diffusion draft) on one H100. Single-stream code: 83 → 357 tok/s (~4.3×). Single-stream prose: 83 → 124 tok/s (~1.5×). Batch 8, code: 588 → 583 tok/s (~break-even). Batch 8, prose: 588 → 393–421 tok/s (~0.7×, slower). Huge single-stream wins where draft alignment is high; negligible or negative once batching saturates compute.

Analogy ❝The cook's apprentice scribbles a guess of the next few bites before the head cook checks the ticket. When the order is a familiar one — the code-shaped dishes the apprentice has seen a hundred times — almost every guess is right, and the head cook waves four bites through at once. On an oddball custom order, the guesses miss, the head cook re-does them anyway, and the scribbling was wasted breath. And during a dinner rush, with every burner already lit, there's no idle hand to do the guessing — the kitchen was already flat out.

Speedup follows acceptance rate: ~82% on code gives ~4 bonus tokens/pass and ~4.3× single-stream; low acceptance or high batch concurrency erases the win — even slows you down.

06

On your cluster

TL;DR · Qwen3.6 ships a built-in MTP self-speculation head (mtp_num_hidden_layers: 1), but the server has no --speculative-* flag set, so it's dormant. Turn it on to speed single-stream decode.

You may already be carrying a draft model and not using it. The target's own config has a multi-token head baked in — it just needs a flag to wake up.

The MTP head is already in the config

Qwen3.6's config includes mtp_num_hidden_layers: 1 — a built-in Multi-Token Prediction (MTP) self-speculation head. It's a draft mechanism living inside the target model itself, ready to propose tokens.

But it's dormant

The server currently runs with no --speculative-* flag enabled, so the MTP head sits idle. Plain autoregressive decode is what you're getting today.

The lever

Enabling MTP speculation speeds single-stream decode. At high batch, the gains shrink — because batching already uses the idle compute the speculation was going to spend.

In the model config — the dormant draft head

// Qwen3.6 config.json (excerpt)
{
  "mtp_num_hidden_layers": 1   // built-in MTP self-speculation head
}

Wake it up — enable speculation at serve time

vllm serve Qwen3.6-27B-FP8 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Keywords — tap to unfold the plain meaning

Check yourself

Speculative decoding speeds up decode by… (what exactly does the draft do, and what does the target do with it)?
Why is the output mathematically identical to normal decoding — what mechanism guarantees that?
Why do the gains shrink under high batch concurrency?
Name the three places a draft can come from.

Speculative decoding uses a cheap draft to guess K tokens, then verifies all K in one target pass and keeps the correct prefix — idle compute does the verify, rejection sampling keeps it identical. Your Qwen3.6 has a dormant MTP head waiting for a --speculative-* flag.