The Forward Pass & Sampling — Inference Engineering

01

The forward pass output: one distribution

TL;DR · One forward pass runs the tokens through every transformer block to produce logits — one raw score per vocabulary token — and softmax turns those into a single distribution over the next token.

Today's win: you'll predict what a forward pass outputs and how decoding turns it into the token you get. Everything in this lesson is the knobs you set per request — but first, what is the pass actually handing you?

Blocks in, logits out

The transformer stack pushes the input vectors through 64 blocks of attention and feed-forward layers. Out the top comes a logit for every token the model knows — for Qwen that's 248,320 raw scores, one per vocabulary entry.

Softmax makes it a distribution

Those logits are just unbounded numbers. Softmax squashes the whole list into probabilities that are all positive and sum to 1 — "one distribution over the next token." That distribution is the real output of the pass.

The pass is the same every time

This is the forward pass from Lesson 04's pipeline, now zoomed in on its tail end. The blocks don't decide a word; they only score every candidate. Choosing a word is a separate step — and that step is what the rest of this lesson is about.

Keywords — tap to unfold the plain meaning

The pass ends in a distribution, not a word. 248,320 logits in, one set of probabilities out — every later knob just reshapes or trims this.

A forward pass through 64 blocks outputs 248,320 logits; softmax makes them one distribution over the next token. The pass scores, it doesn't pick.

02

Greedy decoding: always take the top

TL;DR · The simplest decoder picks the single highest-probability token (argmax). It's deterministic and reproducible — and can be repetitive.

You have a distribution. The most obvious thing to do with it is also the first decoding strategy: just take the winner, every time.

Argmax, nothing more

Greedy decoding picks the single highest-probability token — the argmax of the distribution. No dice, no randomness: the same prompt always yields the same next token.

Deterministic and reproducible

Because it never rolls a die, greedy is deterministic: identical input gives identical output every run. Great for reproducible evals — the trade-off is that it can read as repetitive, always reaching for the safe favorite.

The worked example

For the prompt "Kubernetes pods are scheduled by the", greedy always returns " kube" — the start of "kube-scheduler", the top of the distribution.

The model's raw distribution — Qwen3.6-27B-FP8, prompt: "Kubernetes pods are scheduled by the"

" kube"48%

" scheduler"23%

" K8s"15%

Greedy ignores the spread entirely and takes " kube" — the argmax — no matter what the lower bars say.

Keywords — tap to unfold the plain meaning

Analogy · the pantry ❝Greedy is the line cook with one rule: always cook the single most-ordered dish. Safe, predictable, a little boring. The ticket says the same thing, the cook reaches for the same plate every single time — never a surprise, never a wrong turn, but never anything new either.

Greedy decoding = argmax: always the top token, deterministic and reproducible, sometimes repetitive. Same prompt in, same token out.

03

Temperature reshapes the distribution

TL;DR · Temperature reshapes the distribution before sampling: low values sharpen toward the top token (safe, peaky); high values flatten it (creative, chaotic). Temperature 0 = greedy = deterministic.

Greedy is one extreme. Temperature is the dial that lets long-shots back into the running — or shuts them out entirely — before any die is rolled.

One dial, two directions

Temperature reshapes the probability distribution before sampling. Low values sharpen it toward the top token — safe and peaky. High values flatten it — creative and chaotic, letting unlikely tokens win more often.

The same prompt, three temperatures

Real Qwen results on the kube prompt: at T=0.1 it's ~100% " kube" (greedy-like); at T=1.0 the spread is 48% / 23% / 15%; at T=2.0 it flattens to 29% / 20% / 16% — the long-shots are catching up.

The zero special case

The key principle: Temperature 0 = greedy = deterministic. Turning temperature all the way down collapses sampling back into always-take-the-top — no randomness left at all.

Real Qwen output, same prompt, three temperatures — watch the top token's lead shrink

T=0.1 " kube"~100%

T=1.0 " kube"48%

T=1.0 " scheduler"23%

T=1.0 " K8s"15%

T=2.0 " kube"29%

T=2.0 " scheduler"20%

T=2.0 " K8s"16%

As T rises from 0.1 → 1.0 → 2.0 the favorite's lead collapses from ~100% to 48% to 29% — that's the distribution flattening.

Keywords — tap to unfold the plain meaning

Math, decoded — temperature-scaled softmax

p_i = e^{(z_i / T)}Σ_j e^{(z_j / T)}

z_ithe raw logit the model gives token i — any real number
Ttemperature: divide every logit by it before exponentiating
T < 1small T blows the gaps up — the top logit pulls far ahead, distribution sharpens
T > 1large T shrinks the gaps — scores converge, distribution flattens
T → 0the top token's share approaches 1: this is exactly greedy / argmax

Dividing logits by T before softmax is the whole trick: T below 1 sharpens toward the favorite, T above 1 flattens toward the long-shots, and T=0 collapses to greedy.

Analogy · the pantry ❝Temperature is how loaded the dice are. Near 0 they always land on the favorite; crank it up and long-shots start winning. The cook is still reading the same ticket, but now the choice of next ingredient is decided by dice — and temperature is the weight baked into them.

Temperature divides logits by T before softmax: low = sharp and safe, high = flat and creative, and T=0 = greedy = deterministic.

04

Top-k and top-p trim the unreliable tail

TL;DR · Top-k keeps only the k highest-probability tokens; top-p (nucleus) keeps the smallest set summing to probability p — both cut the unreliable tail so high temperature can't pull in garbage.

Flatten the distribution with temperature and you wake up a long tail of nonsense tokens. Top-k and top-p are the guardrails that throw that tail away before you sample.

A 248,320-token tail

The distribution covers all 248,320 tokens, and the bottom of it is unreliable — tiny probabilities on tokens that would wreck the output. Raise temperature and those garbage tokens get a real shot at being sampled. Trimming the tail prevents that.

Top-k: keep the best k

Top-k retains only the k highest-probability tokens and discards the rest, then samples among the survivors. A fixed-size cut: exactly k candidates, regardless of how peaky or flat the distribution is.

Top-p: keep the nucleus

Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities sum to p (typically 0.9). An adaptive cut: few candidates when the model is confident, more when it's unsure.

Keywords — tap to unfold the plain meaning

Bars sorted high to low. Top-k keeps a fixed count of leftmost bars; top-p keeps the leftmost bars that together reach p. Everything past the cut is thrown away before sampling.

Analogy · the pantry ❝Even with loaded dice, you don't want the cook reaching into the very back of the pantry for something spoiled. Top-k says "only the front k jars are in play." Top-p says "only the jars you'd actually grab nine times out of ten." Either way, the spoiled stuff at the back of the shelf never makes it onto the plate.

Top-k keeps a fixed k tokens; top-p keeps the smallest set summing to p (~0.9). Both cut the unreliable tail so temperature can't sample garbage.

05

Setting the knobs in production

TL;DR · temperature, top_p, and top_k are per-request knobs in the API call; vLLM applies them before sampling each token. Reproducible evals → temperature 0; chat/creative → ~0.7 + top_p 0.9.

These aren't model settings baked in at load time. They ride along with each request, so two callers hitting the same server can get deterministic or wild output — their choice, per call.

Per-request, not per-model

Set temperature, top_p, and top_k per call in the API request. The same loaded weights serve everyone; these knobs change only how this request's distribution is reshaped and trimmed.

vLLM applies them before sampling

vLLM reads these fields and applies them before sampling each token — temperature reshapes, top-k/p trim, then it draws. Every generated token in the request goes through the same configured pipeline.

Two sane defaults

The recommendation: for reproducible evals use temperature 0 (= greedy, deterministic); for chat / creative use ~0.7 + top_p 0.9 — enough variety to feel alive, with the tail still trimmed.

Keywords — tap to unfold the plain meaning

SRE note Sampling params live in the request body, not the server config — so determinism is a per-call property, not a deployment-wide one. For an eval that must reproduce byte-for-byte, send temperature: 0; for product chat, temperature: 0.7 + top_p: 0.9. Same weights, same server, different knobs.

temperature / top_p / top_k are per-request knobs vLLM applies before each sample. Evals → temperature 0; chat → ~0.7 + top_p 0.9.

06

On your cluster

TL;DR · Pull the raw logits with a one-token request and logprobs, and you see the distribution this whole lesson described — before any knob commits to a word.

You don't have to trust the bar charts. One request, capped at a single token, hands you the model's hand directly.

Cap at one token, ask for logprobs

Set max_tokens to 1 and logprobs to 8. The server runs exactly one forward pass and returns the eight most likely next tokens with their log-probabilities — the raw distribution from Station 01, before any decode step commits.

Pull the raw logits

curl .../v1/completions ... "max_tokens":1, "logprobs":8

Cluster note Tested on Qwen3.6-27B-FP8. The logprobs: 8 field returns the eight most likely next tokens with their scores — the same " kube" / " scheduler" / " K8s" spread you saw at T=1.0. Set max_tokens: 1 so the server stops after a single forward pass and you see the distribution unmuddied by later steps.

Keywords — tap to unfold the plain meaning

Check yourself

Finish the sentence for a colleague: "A forward pass gives us a distribution; we get one token by…"
What does temperature do to the distribution, and what is temperature 0 equivalent to?
Top-k vs top-p — which keeps a fixed number of tokens, and which keeps a fixed amount of probability?

A forward pass gives a distribution; you get one token by decoding it — greedy argmax, or sampling with temperature reshaping it and top-k/p trimming the tail. Those are your per-request knobs.