Inside a Transformer Block — Inference Engineering

01

The residual stream is the main path

TL;DR · Each token is a vector riding a shared residual stream; every sublayer reads it, computes a change, and adds that change back — it never overwrites.

A transformer block isn't attention by itself. It's a complete repeated unit: a main path with two specialists writing onto it, wrapped in normalization. Once you see the residual stream, the rest of the block falls into place.

One vector per token, on a shared path

Each token begins as a vector on the residual stream — the block's main path. A sublayer reads the current state, computes a change, and adds that change back. The stream is never replaced; it's refined.

Normalize before, not after

A modern decoder block is pre-norm: it applies RMSNorm (or older LayerNorm) to the stream before each sublayer, so the sublayer sees a clean, scaled input. The raw stream itself flows past, untouched, to receive the addition.

Two sublayers per block

Each block holds two: attention (mixes across tokens) and the MLP (transforms each token alone). Each is wrapped norm → sublayer → add. Stack many such blocks and the same shape repeats all the way up.

Keywords — tap to unfold the plain meaning

Math, decoded — the residual update

x ← x + Sublayer( RMSNorm(x) )

xthe token's current vector on the residual stream — the running state
RMSNorm(x)rescale x to a stable size before the sublayer reads it (pre-norm)
Sublayer(·)attention or the MLP — computes a change, not a replacement
x + …add the change back onto the stream; the original x is preserved

This same shape runs twice per block (once for attention, once for the MLP) and repeats in every block up the stack. The add is what lets a deep model refine instead of overwrite.

The stream flows straight through; each sublayer branches off a normalized copy, computes a change, and adds it back. Same shape, twice per block, every block.

The residual stream is the main path. Each sublayer reads a normalized copy, computes a change, and adds it back — refine, never overwrite.

02

Attention mixes information across tokens

TL;DR · Attention is the cross-token step: each position forms Q, K, V from its normalized state and gathers information from earlier positions only.

This is the one sublayer where tokens are allowed to look at each other. Everything else in the block treats each token in isolation — attention is the mixing.

Q, K, V come from the stream

The normalized token states form Q (query), K (key), and V (value). Each position uses its query to score every key, then takes a weighted blend of the values — that blend is the change added back to the stream.

Causal means look-back only

Causal attention lets each position gather information from earlier positions, never later ones. That mask is what makes generation work left-to-right: a token can only attend to what's already been written.

Why it costs at decode time

Because each new token attends to every prior token, attention is where past keys and values must be kept around. That stored K/V traffic is what the KV cache (covered later) exists to manage as context grows.

Keywords — tap to unfold the plain meaning

Analogy ❝One kitchen station has two specialists. Attention is the researcher: before adding anything to the worktop, they read back through every earlier line of the order ticket — never the lines not yet written — and gather the context that matters for this item. The residual stream is the shared worktop; the researcher sets their findings down onto it for the next specialist to build on.

Attention forms Q, K, V from the normalized stream and blends values from earlier positions only. It's the block's one cross-token step.

03

The MLP transforms each token independently

TL;DR · The MLP expands each token to a wider dimension and projects it back — no cross-token mixing. Qwen/Llama blocks gate it with SwiGLU.

After attention has gathered context, the MLP does the per-token thinking. It's the second specialist, and it never looks sideways — it works on one token at a time.

Expand wide, then project back

The MLP (feed-forward network) expands each token into a wider intermediate dimension and projects it back to the model width. In Qwen3.6-27B that path is 5,120 → 17,408 → 5,120.

Gated SwiGLU adds a learned gate

Qwen/Llama-style blocks commonly use gated SwiGLU, which adds a learned gate that scales the intermediate activation element-by-element before projecting back. The gate decides how much of each wide feature to let through.

Where the parameters and traffic live

These large matrices hold a major share of dense-model parameters and of decode-time weight traffic — the bytes read from memory each step. Per-token compute is cheap; moving those weights is what dominates.

Keywords — tap to unfold the plain meaning

Math, decoded — gated SwiGLU MLP

MLP(x) = ( SiLU(xW_gate) ⊙ (xW_up) ) W_down

xW_upexpand the token from width 5,120 up to the intermediate 17,408
xW_gatea second wide projection that becomes the learned gate
SiLU(·)the smooth activation that shapes the gate (the "Swi" in SwiGLU)
⊙element-wise multiply: the gate scales each wide feature individually
W_downproject the gated 17,408-wide vector back down to 5,120

Up-project to a wider space, gate it element-by-element, then project back. No token ever sees another here — the MLP is purely per-token.

Analogy ❝The MLP is the prep cook: the second specialist at the station, who takes each item on its own and transforms it independently. They never glance at the other orders — just take what's on the worktop, work it wide, and set the result back down onto the same shared residual stream the researcher used.

The MLP expands each token (5,120 → 17,408 → 5,120), gates it with SwiGLU, and projects back — per-token only, and the bulk of dense weight traffic.

04

The LM head closes the stack

TL;DR · After all 64 blocks refine the stream, one final projection — the LM head — scores the whole vocabulary into logits. Then the next token needs the full trip again.

All the refinement happens before any word is scored. Only at the very top, once, does the model turn a polished vector into a guess over every possible token.

All refinement, then one scoring

The stack stacks 64 blocks of norm → attention → add → norm → MLP → add. Every refinement happens inside those blocks; the LM head only runs at the end, projecting the final residual vector to one logit per vocabulary token.

The vocabulary projection is huge

The LM head is a matrix of size hidden × vocabulary — here 5,120 × 248,320. That single projection is one of the largest matrices in the model and produces the distribution you sampled back in earlier lessons.

Every next token is a full trip

Generating each next token requires another full trip through the entire stack — all 64 blocks, then the head — because the residual stream must be rebuilt from the now-longer sequence. There's no shortcut around the depth.

Config note Qwen3.6-27B: 64 layers · hidden size 5,120 · FFN size 17,408 · vocabulary 248,320 tokens. The MLP path is 5,120 → 17,408 → 5,120 per block, and the LM head projects 5,120 → 248,320 once at the top. Those large matrices hold a major share of dense-model parameters and decode weight traffic.

Keywords — tap to unfold the plain meaning

The block is the repeated unit; the LM head runs only once, at the top. All 64 blocks must run again for the next token.

64 blocks refine, then one LM head scores 248,320 tokens. Every next token is another full trip through the entire stack.

05

On your cluster

TL;DR · Read the model config off the served model and you'll see the same numbers — layers, hidden, FFN, vocab — that define the repeated block.

The block isn't an abstraction. Every figure in this lesson is sitting in the config of the model already running on the box.

The config names the block

The model's config exposes the exact shape of the repeated unit: num_hidden_layers, hidden_size, intermediate_size, and vocab_size. Those four numbers fully describe how one decoder block is wired and how many times it stacks.

Read the served model's config on the 4×H100 box

curl -s localhost:8000/v1/models | jq '.data[0].id'
# Qwen3.6-27B-FP8

jq '{layers:.num_hidden_layers, hidden:.hidden_size,
    ffn:.intermediate_size, vocab:.vocab_size}' config.json
# { "layers": 64, "hidden": 5120, "ffn": 17408, "vocab": 248320 }

Cluster note Read 2026-06-19 from the Qwen3.6-27B-FP8 config on the 4×H100 box: 64 layers, hidden 5,120, FFN 17,408, vocab 248,320. The MLP weights (5,120 × 17,408) and the LM head (5,120 × 248,320) are the largest matrices — and the bytes they pull from memory each decode step are what dominate per-token cost.

Keywords — tap to unfold the plain meaning

Check yourself

What does a sublayer do to the residual stream — replace it, or add a change onto it?
Which sublayer mixes information across tokens, and which one works on each token alone?
In a pre-norm block, does normalization happen before or after the sublayer?
What does the LM head produce, and how many times does it run per token compared to a decoder block?

The repeated block is right there in the config: 64 layers, hidden 5,120, FFN 17,408, vocab 248,320. The block is the model.