The Autoregressive Loop & Cost — Inference Engineering

01

A 40-token answer is 40 forward passes

TL;DR · You now have every piece — tokenize → embed → attention → forward pass → decode. Stack them in a loop: one answer of 40 tokens needs 40 forward passes in a row.

Today's win: you'll predict the autoregressive loop, why each token costs more than the last, and the fixes that flatten that staircase — the bridge into the inference runtime.

The whole machine, in a loop

You have every piece now: tokenize → embed → attention → forward pass → decode. A single forward pass produces one token. To write more, you run the whole stack again with the new token appended.

One pass per output token

This is the autoregressive loop: each generated token feeds back in as input for the next step. A 40-token answer is not one big computation — it's 40 forward passes in a row, each waiting for the one before it.

Strictly sequential

You can't run step 5 until step 4 has chosen its token, because step 5's input is step 4's output. That serial dependency is why generation feels like it streams out word by word rather than appearing all at once.

Keywords — tap to unfold the plain meaning

Each pass writes one token, appends it, and loops. The dashed return path is what makes the loop sequential — and what the rest of this lesson is about.

Generation is the whole stack run in a loop: one forward pass per output token, each waiting on the last. A 40-token answer is 40 passes.

02

Each step re-reads everything — the cost staircase

TL;DR · Naively, every step re-reads the entire sequence so far. Step 1 processes a short sequence; step 40 processes 40+ tokens. Work per step grows with length — the cost staircase.

The loop looks innocent, but it hides the central problem of serving: the work each step does grows as the answer grows. Every optimization later is a fight against this one staircase.

Work grows with the sequence

To predict the next token, the naive loop re-reads the entire sequence so far. Step 1 processes a short sequence; step 40 processes 40+ tokens. The work per step grows with length — this is the cost staircase.

The first token is the cheap one

Early steps have little history to re-read, so they're fast. Later steps drag the whole accumulated context back through the model every single time, so each token is a little pricier than the one before it.

Long outputs are doubly expensive

A long answer doesn't just cost more because there are more tokens — each of those later tokens is also individually slower. The two effects compound, which is what makes the total scale faster than linear.

Keywords — tap to unfold the plain meaning

Each bar is one generated token. Because every naive step re-reads the whole past, the bars climb — the cost staircase the rest of the runtime is built to flatten.

Math, decoded

work per step ∝ n → total work ∝ n²

nhow many tokens are already in the sequence when this step runs
∝ nnaive per-step work grows in proportion to n — re-read all n prior tokens
∝ n²summing rising per-step costs over the whole output gives quadratic total work

Naively, the work to produce one token grows with the sequence length n, so generating a long answer scales like n² overall. Flattening that growth is the whole job of the inference runtime.

Analogy ❝Picture a line cook who, after plating every single bite, re-reads the entire order ticket from the very top before deciding the next bite — and the ticket keeps getting longer each bite. Bite one is instant. By bite forty they're re-reading forty lines just to add one more. The food isn't harder to cook; the re-reading is what's piling up. Flattening that re-read is the whole game.

Every naive step re-reads the whole sequence so far, so cost per step rises with n and total work scales like n². That's the cost staircase.

03

Measured: GPT-2 went from 64 ms to 147 ms

TL;DR · On plain GPT-2 with no optimizations, the first token took ~64 ms and the 40th took ~147 ms — more than 2× slower, purely because the sequence grew.

The staircase isn't a theory you have to trust. Run a raw model with the optimizations switched off and you can watch each token get measurably slower than the last.

The same loop, timed

On plain GPT-2, generating a 40-token answer means 40 forward passes. Timed individually, the first token took ~64 ms and the 40th took ~147 ms — the cost staircase, in milliseconds.

More than 2× slower

That's more than 2× slower for the last token versus the first, and nothing about the model changed between them. The only difference is that the sequence grew from a handful of tokens to 40+.

Measured on plain GPT-2, no optimizations — same 40-token answer, per-token forward-pass time

token 164 ms

token 40147 ms

More than 2× slower, purely because the sequence grew from a few tokens to 40+. The weights never changed — only the amount of past that each step re-reads.

Keywords — tap to unfold the plain meaning

Plain GPT-2: first token ~64 ms, 40th token ~147 ms — more than 2× slower, with no model change. Just a longer sequence to re-read.

04

Three fixes that flatten the staircase

TL;DR · KV cache stops the re-reading, batching shares each weight-read across requests, and speculative decoding writes several tokens per pass. Together they flatten the staircase.

The whole inference runtime you're about to study is essentially these three ideas, dressed up. Meet them once here, in plain form, and the next dozen lessons will feel like footnotes.

Fix 1 — KV cache

Instead of recomputing the past on every step, the KV cache stores past tokens' Keys/Values so each step avoids recomputing them — each step then does ~constant work instead of re-reading the whole sequence.

Fix 2 — Batching

Batching means one weight-read serves many requests at once, amortizing the expensive part. Loading the weights is the costly move; if many requests ride along on a single read, the cost is shared across all of them.

Fix 3 — Speculative decoding

Speculative decoding uses a cheap model to guess several tokens, then verifies them in one pass of the big model. When the guesses are right, you get multiple tokens for the price of roughly one forward pass.

Keywords — tap to unfold the plain meaning

Why batching helps Decode is memory-bound: most of the time per step goes to reading the weights out of GPU memory, not to the arithmetic. That's why batching works — one weight-read can serve many requests, so the slow part is paid once and shared. Hold this; the roofline lesson makes it exact.

Analogy ❝The weights sit in a faraway pantry, and every step the cook has to walk the whole way there to fetch them. KV cache means the cook keeps the prepped ingredients on the counter instead of re-chopping them each bite. Batching means that single long walk to the pantry now carries the order for every table at once, not just one. Speculative decoding means a fast junior cook sprints ahead and plates a few likely bites, and the head cook just glances to confirm them in one look.

KV cache → ~constant work per step. Batching → one weight-read serves many requests. Speculative decoding → several tokens verified in one pass.

05

On your cluster: ~108 gen tok/s, held flat

TL;DR · Your Qwen on vLLM already uses every fix above — which is why it sustains ~108 generation tok/s instead of degrading token-by-token like raw GPT-2.

This is the bridge into the inference runtime. The reason your server doesn't slow down the way raw GPT-2 did is that the staircase has already been flattened — by exactly the three fixes you just met.

The staircase, already flattened

Your Qwen on vLLM already applies all three fixes — KV cache, batching, and speculative decoding. That's why it holds a steady ~108 generation tok/s instead of degrading token-by-token like raw GPT-2.

What the rest of the course unpacks

Every term ahead — prefill vs decode, the KV cache and its memory, PagedAttention, the roofline, speculative decoding — is one of these fixes, taken apart and made precise. You've now seen the shape of all of them.

Keywords — tap to unfold the plain meaning

Cluster note Measured on the 4×H100 box: Qwen on vLLM sustains ~108 gen tok/s. The baseline comparison — plain GPT-2 climbing from ~64 ms to ~147 ms per token — has no KV cache, no batching, no speculative decoding. Same loop, optimizations off vs on.

Check yourself

Finish the sentence for a colleague: "Naive generation gets slower per token because…"
A 40-token answer takes how many forward passes — and why can't they run in parallel?
Name the three fixes that flatten the cost staircase, and say in one line what each one does.

Naive generation slows per token because it's a loop where every step re-reads the whole sequence so far. The fixes are why your Qwen holds ~108 tok/s.