Architecture Variants — Inference Engineering

01

Decoder-only is the serving baseline

TL;DR · Almost every generative LLM server repeatedly runs a causal decoder stack — and that repetition is what sets latency and state management.

Before the variants, fix the shape. The model you serve is one family out of three, and the one you serve was chosen because of how it generates: one token at a time, appended and re-run.

Three families, one winner for generation

Transformers come in three shapes. An encoder-only model produces a representation for classification. An encoder-decoder encodes the input once, then generates step-by-step. A decoder-only model — the serving baseline — is a causal decoder that appends tokens repeatedly.

Why generation picks decoder-only

Most generative LLM servers repeatedly run a causal decoder stack. Those repeated autoregressive passes are what determine latency and state management in language-model serving — so the decoder's shape is the thing the whole runtime is built around.

Keywords — tap to unfold the plain meaning

Of the three transformer families, generative servers run the decoder-only causal stack — and its repeated passes are what set latency and state.

02

What modern LLMs kept from 2017

TL;DR · The 2017 Transformer had an encoder and a decoder; today's GPT/Qwen keep only the right half — same building blocks, a simpler shape.

The architecture you serve is a deliberate trim of the original. Knowing what got cut tells you why generation looks the way it does.

The original two-stack design

The 2017 Transformer had an encoder stack that reads the whole input at once — every token attends to every other — and a decoder stack that generates left-to-right with masked self-attention (which prevents peeking at future tokens) plus cross-attention reading the encoder's output.

Modern decoder-only LLMs

GPT and Qwen keep only the right half — no encoder, no cross-attention. They also use pre-norm (normalize before each sublayer) instead of post-norm. Same building blocks, a simpler shape.

Keywords — tap to unfold the plain meaning

The original had two stacks joined by cross-attention. Modern LLMs drop the encoder entirely and keep the masked decoder — with pre-norm instead of post-norm.

Modern LLMs keep the decoder's masked self-attention, drop the encoder and cross-attention, and normalize before each sublayer (pre-norm).

03

MHA, GQA & MQA change KV width

TL;DR · How many query heads share each K/V head decides how wide your KV cache is — and that width is fixed at training time.

This is the single config knob with the biggest effect on per-request memory. Read the head counts and you can predict the KV cache before you ever start the server.

The three sharing schemes

MHA (Multi-Head Attention) gives every query head its own K/V head. GQA (Grouped Query Attention) shares each K/V head across a group of query heads. MQA (Multi-Query Attention) shares one K/V head across all queries.

Fewer KV heads, smaller cache

Fewer KV heads shrink the KV cache — the per-request attention memory the model builds up while generating (computed in Lesson 14) — and the attention bandwidth. Here "bandwidth" means GPU-memory bandwidth: the speed of reading weights and the KV cache from HBM, not network bandwidth.

You can't change it later

The architecture is fixed at training time. You read the head counts off the config and live with them — there's no flag that converts MHA to GQA on a model that was trained as MHA.

Same eight query heads in all three; only the K/V head count shrinks — 8 → 2 → 1. Fewer K/V records means a narrower KV cache and less HBM bandwidth per step.

Keywords — tap to unfold the plain meaning

Analogy ❝Think of GQA and MQA as shared index-card writers in the kitchen. Many query readers — the line cooks glancing back at what's been ordered — reuse a much smaller pile of K/V index cards instead of each keeping a private set. With MQA there's a single shared card pile for the whole crew. Fewer cards to keep and fewer to re-read from the faraway pantry (HBM) each step: that's why a narrow KV cache serves faster.

Cluster note Lab finding: the Qwen3.6 config demonstrates GQA directly — 24 query heads share 4 KV heads, cutting KV width by 6× versus full MHA. Read those two numbers off any config and you can predict the per-request KV memory before launch.

Math, decoded

KV width ∝ n_kv → n_qn_kv = 244 = 6× saving

n_qnumber of query heads — here 24 (the readers that ask "what's relevant?")
n_kvnumber of K/V heads — here 4 (the shared key/value records actually stored)
n_q / n_kvthe sharing ratio — how many queries reuse each K/V head: 24 ÷ 4 = 6
∝ n_kvKV cache size scales with K/V heads, not query heads — fewer K/V heads, smaller cache

KV cache width tracks the K/V head count, not the query head count. Qwen3.6's 24:4 ratio means GQA stores one-sixth the K/V data of full MHA — a 6× cut in per-request attention memory and HBM bandwidth.

MHA = one K/V head per query; GQA shares per group; MQA shares one. Fewer K/V heads = smaller KV cache and less HBM bandwidth — fixed at training time.

04

MoE separates total from active parameters

TL;DR · A Mixture-of-Experts model stores many expert MLPs but runs only a routed few per token — so total parameters and active parameters become two different numbers, and both matter.

In a dense model these two numbers are equal, so nobody distinguishes them. MoE splits them apart, and that split is the whole reason MoE changes how you plan memory and compute.

Experts and a router

A MoE block owns many expert MLPs (the feed-forward network, or FFN — also called the MLP — is the big per-token math block in each layer). A router sends each token to a small top-k subset of those experts.

Two numbers, both load-bearing

Total parameters determine storage and placement — every expert is stored whether or not it fires. Active parameters better approximate per-token expert compute — only the routed top-k actually run. Both matter for inference.

Dense vs MoE

In a dense MLP every token goes through the same dense MLP, all weights active, so total parameters = active parameters — simple, predictable compute per token. In MoE, all experts are stored but only the routed top-k are computed, so capacity grows faster than per-token arithmetic.

Dense: one MLP, every weight active, total = active. MoE: four experts stored (total), only the routed top-k run (active) — so storage can balloon while per-token math stays lean.

Keywords — tap to unfold the plain meaning

Analogy ❝A dense kitchen sends every order through the same crew — one team, every dish. An MoE kitchen owns many specialist stations but dispatches each ticket to only a few. You still pay rent on every station (that's total parameters — storage), but each plate only fires up the stations it's routed to (that's active parameters — the per-token cooking). The pantry is huge; each order only walks to a couple of shelves.

Cluster note Lab note: the lab's model is not a standard top-k MoE, so its expert-routing claims are taught as a separate architecture case rather than read straight off the served config.

MoE stores many experts (total params = storage) but a router fires only top-k per token (active params = compute). Dense makes the two equal; MoE splits them — both matter.

05

Routing creates systems costs

TL;DR · Routing tokens to experts isn't free — experts get imbalanced, tokens cross GPUs, and capacity limits drop work, so tail latency rises.

The split between total and active parameters looks like a pure win until you serve it across GPUs. Then routing turns into a communication-and-balancing problem that the dense path never had.

Three failure modes

Routing introduces three ways things go wrong: experts can be imbalanced (some get far more tokens than others); tokens may cross GPUs to reach their expert; and capacity limits can drop or reroute work when an expert is full.

Expert parallelism's two extra phases

Under expert parallelism, a dispatch phase combines local and cross-rank token routes, then a return + combine phase sends expert outputs back to each origin rank. That's two extra collective communication transfer-and-synchronization phases the dense path never pays.

The slowest route sets the pace

Uneven routing also creates load imbalance, so inference waits for the slowest transfer or the overloaded expert and tail latency rises. Idle experts cannot hide the slowest route; communication and queues raise latency.

Dispatch sends tokens to whichever rank holds their expert; return + combine brings the outputs back. Two collective phases plus uneven load mean the slowest route sets the step's pace.

Keywords — tap to unfold the plain meaning

Analogy ❝Picture one narrow road from the kitchen to the specialist stations. Every routed ticket has to travel out (dispatch) and the finished plates have to travel back (return + combine). If one station gets slammed with orders while another sits idle, the whole service waits on the busy one — the idle cook can't plate someone else's order. The road and the queue, not the cooking, are what pile up.

Routing adds a dispatch and a return+combine phase, plus load imbalance. Inference waits for the slowest route or busiest expert — so tail latency, not average compute, is the cost.

06

On your cluster

TL;DR · Read the head counts and expert fields straight off the model config — they tell you the KV width and the total-vs-active split before you ever launch.

The whole lesson reduces to a config-reading skill. Open the JSON, find five fields, and you can predict memory, compute, and serving complexity without guessing.

The fields that decide everything

Look for the query-head and KV-head counts (the GQA/MQA ratio that sets KV width) and any expert/top-k fields (the total-vs-active split). On the lab box, the config shows 24 query heads sharing 4 KV heads — GQA, a 6× KV cut.

Read the config on the 4×H100 box

cat Qwen3.6-27B-FP8/config.json | jq '{
  num_attention_heads,   # query heads  → 24
  num_key_value_heads,   # K/V heads    → 4   (GQA, 6x KV cut)
  num_experts,           # MoE total experts (if MoE)
  num_experts_per_tok    # routed top-k per token (active)
}'

Cluster note On the 4×H100 box, num_attention_heads: 24 with num_key_value_heads: 4 confirms GQA — a 6× narrower KV cache than full MHA, exactly the ratio from Station 03. If num_experts appears, total params (storage) and active params (top-k compute) diverge — and remember the lab model is not a standard top-k MoE, so treat its routing as a separate case.

Keywords — tap to unfold the plain meaning

Check yourself

Which transformer family do generative LLM servers run — and what did modern LLMs drop from the 2017 design?
In MHA, GQA, and MQA, what changes — and which count (query heads or K/V heads) sets the KV cache width?
In an MoE model, what's the difference between total and active parameters, and which one drives storage versus per-token compute?
Name the two extra communication phases expert parallelism adds, and why uneven routing raises tail latency.

The config is the answer key: num_key_value_heads sets KV width, expert fields set the total-vs-active split. On the lab box, 24:4 = GQA, a 6× KV cut.