The complete repeated unit, not attention alone.
This lesson opens up a single decoder block: its residual stream, attention, MLP (the feed-forward network, or FFN, the big per-token math block in each layer), and final LM head (the output, or LM, head is the final layer that scores every token in the vocabulary). To see where this one block sits in the whole model, zoom back out to Lesson 7 · the original Transformer →
One kitchen station has two specialists: a researcher who gathers context (attention) and a prep cook who transforms each item independently (MLP). The residual stream is the shared worktop both add their results back onto.
Each token begins as a vector on the residual stream. A modern decoder block usually applies RMSNorm or LayerNorm (normalization rescales each vector to a controlled size so a deep stack of layers stays numerically stable; pre-norm does it before each sub-layer) before a sublayer, computes a change, then adds that change back. Residual connections preserve information and make many blocks trainable.
Normalized token states form Q, K, and V. Causal attention lets each position gather information from earlier positions. Its KV states are the part cached across decode steps.
After another norm, the feed-forward network (the MLP and the FFN are the same per-token math block) expands each token into a wider intermediate dimension and projects it back. Qwen/Llama-style blocks commonly use gated SwiGLU, which adds a learned "gate" that scales the intermediate activation (the intermediate vector the layer outputs). These large matrices hold a major share of dense-model parameters and decode weight traffic.
After dozens of blocks, a final norm and vocabulary projection produce one logit per token (logits = the raw per-token scores, before softmax turns them into probabilities; see Lesson 8). Sampling chooses the next ID; the same stack runs again for the next decode step.
Qwen3.6-27B's served config has 64 layers, hidden size 5,120, FFN size 17,408, and a 248,320-token vocabulary. Those dimensions make the repeated block matrices, not the tokenizer, the dominant weight footprint.
The complete repeated unit, not attention alone.
This lesson opens up a single decoder block: its residual stream, attention, MLP (the feed-forward network, or FFN, the big per-token math block in each layer), and final LM head (the output, or LM, head is the final layer that scores every token in the vocabulary). To see where this one block sits in the whole model, zoom back out to Lesson 7 · the original Transformer →
One kitchen station has two specialists: a researcher who gathers context (attention) and a prep cook who transforms each item independently (MLP). The residual stream is the shared worktop both add their results back onto.
| The residual stream is the main path | Each token begins as a vector on the residual stream |
| Attention mixes information across tokens | Normalized token states form Q, K, and V |
| The MLP transforms each token independently | After another norm, the feed-forward network expands each token into a wider intermediate dimension and projects it back |
| The LM head closes the stack | After dozens of blocks, a final norm and vocabulary projection produce one logit per token |
Each token begins as a vector on the residual stream. A modern decoder block usually applies RMSNorm or LayerNorm (normalization rescales each vector to a controlled size so a deep stack of layers stays numerically stable; pre-norm does it before each sub-layer) before a sublayer, computes a change, then adds that change back. Residual connections preserve information and make many blocks trainable.
Normalized token states form Q, K, and V. Causal attention lets each position gather information from earlier positions. Its KV states are the part cached across decode steps.
After another norm, the feed-forward network (the MLP and the FFN are the same per-token math block) expands each token into a wider intermediate dimension and projects it back. Qwen/Llama-style blocks commonly use gated SwiGLU, which adds a learned "gate" that scales the intermediate activation (the intermediate vector the layer outputs). These large matrices hold a major share of dense-model parameters and decode weight traffic.
After dozens of blocks, a final norm and vocabulary projection produce one logit per token (logits = the raw per-token scores, before softmax turns them into probabilities; see Lesson 8). Sampling chooses the next ID; the same stack runs again for the next decode step.
Qwen3.6-27B's served config has 64 layers, hidden size 5,120, FFN size 17,408, and a 248,320-token vocabulary. Those dimensions make the repeated block matrices, not the tokenizer, the dominant weight footprint.