Embeddings — Inference Engineering

01

A bare ID becomes meaning

TL;DR · Tokenization left you with bare integers. An embedding looks each integer up in a table to fetch a learned dense vector — the token's place in a meaning-space.

Last lesson turned text into token IDs. But an integer like 9743 means nothing to the math inside the model. Embeddings are the step that gives each ID an actual meaning the network can compute with.

An integer carries no meaning

A token ID is just an index — a row number. The number 9743 isn't bigger or closer to anything; it's a name, not a measurement. The model can't multiply a row number and learn from it.

The lookup hands back a vector

An embedding is a learned dense vector, fetched by a table lookup, that places the token in a meaning-space where related tokens sit near each other. The ID's only job is to point at the right row.

"Dense" means every slot is used

A dense vector is a list of real numbers with no empty slots — unlike a one-hot row that's all zeros but one. Every dimension carries a little of the token's learned meaning.

Keywords — tap to unfold the plain meaning

Analogy ❝Think of the line cook again. The order ticket just says item numbers — "#9743, #112, #88." The number itself tells you nothing about the dish. So the cook flips to that row in the pantry binder and reads the full description: ingredients, where it sits, what it pairs with. The ID is the row number; the embedding is the rich entry the cook actually reaches for.

A token ID is a row number with no meaning. The embedding lookup swaps it for a learned dense vector — the token's position in meaning-space.

02

The embedding table is just rows × dimensions

TL;DR · The embedding table has one row per vocabulary token and one column per hidden dimension. GPT-2 is 50,257 × 768; Qwen3.6 is 248,320 × 5,120.

There's no magic in the lookup — it's a giant grid of learned numbers. Knowing its exact shape tells you both how rich each token's meaning is and how much memory the table eats.

One row per token

The embedding table has exactly as many rows as the vocabulary has tokens. To embed token ID i, you read row i. That's the whole operation — a row read.

One column per dimension

The number of columns is the hidden dimension — how many numbers describe each token's meaning. GPT-2 uses 768; Qwen3.6 uses 5,120. Wider means each token carries more learned nuance.

Same shape, very different scale

GPT-2's table is 50,257 × 768. Qwen3.6's is 248,320 × 5,120 — far more tokens, each described by a far wider vector. That single grid is the real input to the transformer blocks.

The ID picks a row; the row is the token's dense vector. Width = hidden dimension (5,120 for Qwen3.6, 768 for GPT-2); height = vocabulary size.

Keywords — tap to unfold the plain meaning

Cluster note On the 4×H100 box, Qwen3.6's embedding table alone is 248,320 × 5,120 learned values — a substantial slab of the weights that must live in GPU memory before a single token is served. Wider hidden dimensions and bigger vocabularies make this table grow in both directions at once.

The embedding table is rows × dimensions: one row per token, one column per hidden dimension. GPT-2 is 50,257 × 768; Qwen3.6 is 248,320 × 5,120.

03

Meaning alone forgets the order

TL;DR · The lookup gives each token its meaning but says nothing about where it sits. Order has to be added back as position information, or "dog bites man" and "man bites dog" look identical.

A pile of meaning-vectors is just a set — no first, no last. But language is sequential. The model needs a second signal that says where in the line each token stands.

The same word, two positions

Look up "dog" and you get the same vector whether it's the first word or the last. Without extra signal, the model sees an unordered bag of meanings and can't tell "dog bites man" from "man bites dog".

Position information fills the gap

Position information is a second ingredient added alongside meaning so each token knows where it sits. It's what preserves sequence order through the rest of the network.

Two families of approach

GPT-2 stores a learned vector for each slot (a learned position embedding, the wpe table) and adds it. Modern models like Qwen and LLaMA instead use RoPE, applied on the fly — the next station.

Keywords — tap to unfold the plain meaning

Analogy ❝The pantry binder tells the cook what each item is, but not the order they were called out. If the ticket reads "sauce, then sear" versus "sear, then sauce," the ingredients are identical — only the sequence differs, and it changes the whole dish. Position information is the cook noting which step came first, so the binder entries don't collapse into one undifferentiated pile.

The meaning lookup is order-blind. Position information is the second ingredient that restores sequence — GPT-2 adds a learned vector; modern models use RoPE.

04

RoPE rotates the vector instead of adding to it

TL;DR · Instead of adding a learned position vector, RoPE rotates each token's vector by an angle set by its position — so position isn't stored per-row, it's applied on the fly.

RoPE is why a model can stretch to a 262,144-token context without storing a position vector for every slot. The position lives in a rotation, not a row.

Rotate, don't add

RoPE (rotary position embedding) rotates each vector by an angle that depends on its position. GPT-2 adds a learned position vector (wpe); RoPE rotates, so nothing position-specific has to be stored per row.

Applied on the fly

Because position is a rotation, it's applied at compute time, not looked up. That's how Qwen3.6 reaches a 262,144-token context without a position table that grows with context length.

The knobs that tune it

Qwen3.6 sets rope_theta to 10,000,000 — the base frequency that controls how fast the rotation turns across dimensions. A partial rotary factor of 0.25 means only a quarter of each vector's dimensions get rotated.

Keywords — tap to unfold the plain meaning

Math, decoded

x′ = R(pos, θ) · x , θ = 10,000,000 , rotate 0.25 · d dims

xthe token's meaning vector straight from the embedding lookup
R(pos, θ)a rotation by an angle set by the token's position and the base θ
poswhere the token sits in the sequence — the bigger the position, the larger the turn
θrope_theta = 10,000,000; the base frequency tuning how fast the rotation turns
0.25 · dpartial rotary factor — only a quarter of the d dimensions are rotated

RoPE multiplies the meaning vector by a position-dependent rotation rather than adding a stored vector. Larger position → larger angle; rope_theta sets the spin rate; the partial factor rotates only a fraction of the dimensions.

Cluster note Qwen3.6 on the 4×H100 box runs RoPE with rope_theta = 10,000,000 and a partial rotary factor of 0.25, supporting a 262,144-token context window. Because position is applied on the fly as a rotation, none of that context length costs a per-row position table — the position table never grows.

GPT-2 adds a learned position vector; RoPE rotates the vector by its position on the fly. That's what lets Qwen3.6 reach a 262,144-token context with no per-row position table.

05

The input vector = meaning + position

TL;DR · Put it together: every token enters the transformer as one vector that combines its meaning (the table lookup) with its position (a learned vector or a RoPE rotation).

This is the single formula the whole lesson builds to — and the exact thing fed into the first attention layer you'll meet next.

The two ingredients combine

The vector that actually enters the transformer is meaning + position: the token embedding from the table, plus position — a learned vector added in (GPT-2) or a RoPE rotation applied on the fly (Qwen/LLaMA).

This is what attention reads

That combined vector is the real input to the transformer blocks. Attention — the next lesson — never sees the bare ID; it sees this meaning-plus-position vector, one per token in the sequence.

Math, decoded

input = meaning + position

inputthe per-token vector that actually enters the transformer block
meaningthe token embedding fetched from the table by the token ID
positiona learned position vector (GPT-2) added in, or a RoPE rotation (Qwen/LLaMA) applied on the fly

Input vector = meaning + position. Meaning is always the table lookup; position is either an added learned vector or an on-the-fly rotation — pick the family and you have the model's input.

Keywords — tap to unfold the plain meaning

Peek at the embedding row on the 4×H100 box

python -c 'from transformers import AutoModel
m = AutoModel.from_pretrained("Qwen3.6-27B")
print(m.embed_tokens.weight.shape)   # torch.Size([248320, 5120])
print(m.embed_tokens.weight[9743])   # the dense vector for token 9743'

Check yourself

How does a bare token ID become something meaningful the model can compute with?
Why does order matter, and what is added to a meaning-only vector to preserve it?
State the formula: input vector = ? + ?

Input vector = meaning + position. The lookup gives meaning; a learned vector or a RoPE rotation gives position. That combined vector is exactly what attention reads next.