What's Inside a Model — Inference Engineering

01

A model is a file of named tensors

TL;DR · A model file is an index plus a blob: named tensors — arrays of floats — with a header listing each one's name, shape, and byte offset.

Last lesson the weights were a black box. Open the file and the mystery dissolves: there's no code in there, just measurements — numbers in named drawers.

Just named arrays of numbers

A tensor is a multi-dimensional array of numbers. A model file is nothing more than a pile of named tensors — for example GPT-2 holds 148 tensors totalling 124 million parameters. No code, no magic: just measurements.

A header, then a blob

The file splits in two. The header is a JSON index: every tensor's name, shape, and byte offset. After it comes the raw data blob — one contiguous run of floats. To find a tensor you read its offset from the header, then jump into the blob.

Tensors have names and shapes

The token-embedding table is named wte.weight with shape [50257, 768] — one 768-number row for each of GPT-2's 50,257 vocabulary entries. The shape tells you the dimensions; the name tells you the role.

Keywords — tap to unfold the plain meaning

The header is a lookup table; the blob is the data it points into. "Loading the model" is just mapping that blob into GPU memory.

A model file is a header (an index of each tensor's name, shape, and byte offset) plus a big blob of raw floats. No code, no magic — just measurements.

02

It's one block, repeated N times

TL;DR · All those tensors form a simple stack: an embedding table, then the same transformer block repeated N times, then a final norm and output head.

148 tensors sounds like chaos. It isn't — it's one small pattern photocopied over and over, bookended by a lookup at the top and a scorer at the bottom.

The stack, top to bottom

Three layers of structure: (1) an embedding table that turns each token ID into a vector, (2) the same transformer block repeated N times, and (3) a final norm plus an output head. The middle is just one block, copied.

What lives in one block

Each transformer block holds the attention projections (the Q / K / V / O matrices), a feed-forward network (the FFN, also called the MLP), and a couple of layer norms. Every copy has the same shape; only the numbers differ.

Top lookup, bottom scorer

The embedding table is a lookup: token ID in, vector out. The LM head is the mirror at the bottom — the final layer that scores every token in the vocabulary, producing the logits you met in Lesson 1.

Keywords — tap to unfold the plain meaning

Lookup at the top, scorer at the bottom, and in between the same block stacked N deep. Learn one block and you understand the whole model.

The architecture is an embedding table + one transformer block repeated N times + a final norm and output head. One block, photocopied.

03

Where the parameters actually live

TL;DR · In GPT-2 the parameters split roughly FFN 41% · attention 30% · embeddings 28%. The FFN is the biggest chunk — it's where most "knowledge" is stored.

If you want to know where a model spends its memory, count the parameters per part. One block dominates, and inside the block one piece dominates again.

The FFN is the biggest chunk

Add up GPT-2's parameters and the feed-forward network takes the largest share. It's the big per-token math block in each layer — and it's where most of the model's "knowledge" is stored.

The three-way split

Across the whole model the parameters land at roughly FFN 41%, attention 30%, and embeddings 28%. Attention does the routing; the FFN does the remembering; the embeddings translate in and out.

GPT-2 parameter distribution — measured from 124M parameters across 148 tensors

FFN / MLP41%

attention30%

embeddings28%

The FFN wins — that's the per-token math block where most "knowledge" is stored. Attention is the runner-up; embeddings handle the lookups at each end.

Keywords — tap to unfold the plain meaning

Analogy ❝Think of a line cook's kitchen. The embedding table is the pass where tickets come in and plates go out — translating between the dining room and the line. Attention is the cook glancing back over the whole ticket to decide what matters right now. But the FFN is the deep pantry behind the line: the biggest room, stacked with everything the cook has ever learned to reach for. Most of the kitchen's square footage — and most of its knowledge — lives in that pantry.

GPT-2's parameters split FFN 41% · attention 30% · embeddings 28%. The FFN is the biggest chunk — the pantry where most knowledge is stored.

04

Scaling up: GPT-2 to Qwen3.6-27B

TL;DR · Nothing new appears at scale — the same stack just gets wider and deeper. Qwen3.6-27B is ~220× GPT-2's parameters, but it's still embedding + block ×N + head.

The cluster model is 220 times bigger, yet you already know its anatomy. Every number that grows is one you've already met; the shape is identical.

The same knobs, turned up

GPT-2 has 12 layers, hidden size 768, vocab 50,257, 124M parameters. Qwen3.6-27B reads off its config.json as 64 layers, hidden size 5,120, FFN 17,408, vocab 248,320, 27,000M (27B) parameters — about ~220× the parameters of GPT-2.

Attention gets a head_dim

Qwen3.6 uses 24 query heads and 4 KV heads with a head_dim of 256 — the length of one head's Key/Value vector. Fewer KV heads than query heads is a deliberate memory-saving choice you'll meet again with the KV cache.

The real model is messier — on purpose simplified

The shipped Qwen3.6 is an advanced variant: it mixes linear attention with full attention (full only every 4th layer) and is multimodal (it accepts vision tokens). The lesson teaches the clean version first; the messiness is a later concern.

Keywords — tap to unfold the plain meaning

Math, decoded

params ∝ N_layers × ( h² + h × f ) → 124M → 27,000M ≈ 220×

N_layershow many transformer blocks are stacked: 12 in GPT-2, 64 in Qwen3.6
hhidden size — how many numbers are in each token's vector: 768 vs 5,120
fthe FFN's inner width: 3,072 in GPT-2, 17,408 in Qwen3.6
h²attention projections scale with the square of hidden size — why width is expensive
220×deeper (more layers) and wider (bigger h and f) multiply out to ~220× more parameters

Parameters grow with depth (more blocks) times width (bigger hidden size and FFN). Turn every knob up and GPT-2's 124M becomes Qwen3.6's 27,000M — about 220× — with no new kind of part added.

Cluster note Qwen3.6-27B's numbers come straight from the config.json served on the 4×H100 box: 64 layers, hidden 5,120, FFN 17,408, vocab 248,320, 24 query / 4 KV heads, head_dim 256. The deployed model mixes linear and full attention (full every 4th layer) and handles vision tokens — but the bones are still embedding + block ×N + head.

Scaling adds no new parts — it just turns up depth and width. Qwen3.6-27B (64 layers, hidden 5,120) is ~220× GPT-2 but the same stack.

05

Loading: mapping the blob into GPU memory

TL;DR · "Loading the model" means memory-mapping that float blob into GPU memory. SafeTensors maps in ~2–3s and runs no code; FP8 puts 27B in ~27 GiB.

All this anatomy only matters when the numbers reach the GPU. The format you store them in decides how fast they load — and whether loading can run hostile code.

Loading is just a memory-map

"Loading the model" means mapping the raw float blob into GPU memory — no parsing, no decompression, just pointing the GPU at the bytes. With SafeTensors this memory-maps straight to GPU in ~2–3s, versus 10s+ for old pickle .pt files.

SafeTensors is data-only — and safe

SafeTensors stores data only: there's no code to execute on load. The older pickle format can execute arbitrary code on load — which is exactly why "safe" is in the name. Faster and safer, for the same numbers.

The format sets the size

Each number's precision sets the file size. At FP8 every number is 1 byte (vs 2 bytes for FP16/BF16), so Qwen3.6-27B's weights fit in ~27 GiB — comfortably inside one 94 GiB H100, no tensor-parallel split needed.

Keywords — tap to unfold the plain meaning

Analogy ❝A faraway pantry is useless if it takes ten minutes to walk to. SafeTensors is the pantry built right behind the line: the cook memory-maps it open in seconds and starts plating. The old pickle format is the pantry across town — slow to reach, and worse, anyone could have slipped a booby-trap onto its shelves, because opening it runs whatever code was left inside. Same ingredients, but one pantry is fast and trustworthy and the other isn't.

Cluster note On the 4×H100 box, the FP8 Qwen3.6-27B weights are ~27 GiB and load in ~2–3s via SafeTensors memory-mapping — they fit one 94 GiB H100 with room to spare, so no tensor-parallel split is required just to hold the weights. The same model as a pickle .pt would take 10s+ and run arbitrary code on load.

Loading is memory-mapping the float blob into GPU memory. SafeTensors does it in ~2–3s with no code execution; at FP8 (1 byte/number) 27B fits ~27 GiB.

06

Say it in one sentence

TL;DR · A model file is an index plus a blob of numbers — named tensors memory-mapped into the GPU, shaped as embedding + block ×N + head. No code, no magic.

If you can explain a model to a colleague in one breath, you own this lesson. Here's the sentence, and here's how to check yourself before you move on.

Explain it to a colleague

"A model file is basically an index plus a blob of numbers: named tensors (arrays of floats) memory-mapped into the GPU, organized as an embedding table + one transformer block repeated N times + an output head. No code, no magic: just measurements."

No-peeking recall — what is a model file, concretely?

A model file =
  a header  (an index: each tensor's name, shape, byte offset)
  + a big blob of raw floats

Keywords — tap to unfold the plain meaning

Check yourself

Finish the sentence for a colleague: "A model file is basically…"
What two parts make up the file concretely — what's in the header, and what's in the blob?
What are the three layers of the architecture, top to bottom?
Which part holds the most parameters in GPT-2, and roughly what share?

A model file is an index plus a blob: named tensors memory-mapped into the GPU, shaped as embedding + block ×N + head. No code, no magic — just measurements.