The token is the unit underneath everything
Before the model can predict anything, your text has to become numbers. The token is that number — and it's the currency of the whole rest of this course.
What a token actually is
A token is a piece of text — usually a subword — mapped to an integer ID: its index in the model's fixed vocabulary. The model never sees letters; it sees a short list of these integers and works entirely in number-space.
Why this is Lesson 3
The token is the unit of everything that follows. Prefill processes N tokens, decode emits one per step, the KV cache is stored per token, the context window is a token budget, and the bill is per token. Get the token right and every later cost makes sense.
Keywords — tap to unfold the plain meaning
A token is a chunk of text mapped to an integer ID in a fixed vocabulary. Prefill, decode, KV, context, and cost are all counted in tokens.
The tokenizer: text ↔ integer IDs
There's one component that stands between human words and the model's numbers, in both directions. It's small, deterministic, and easy to forget — until your token counts surprise you.
Encode: text becomes IDs
Tokenization chops your text into pieces from a fixed vocabulary, then replaces each piece with its integer ID — its index in that vocabulary. The component that does this is the tokenizer: text → integer IDs.
Decode: IDs become text
The same tokenizer runs in reverse. After the model emits a token ID, the tokenizer maps it back to its text piece and glues the pieces together — text ← integer IDs. Encode and decode are the two directions of one fixed mapping.
A leading space is part of the token
Pieces carry their leading space. A leading space is shown as Ġ and a newline as Ċ; truly unknown characters fall back to raw UTF-8 bytes. That's why " strawberry" (with a space) and "strawberry" (without) tokenize differently.
Keywords — tap to unfold the plain meaning
The tokenizer maps text → integer IDs and back. Pieces carry leading spaces (Ġ) and newlines (Ċ); the model only ever sees the numbers.
Why subwords won — the three-way trade-off
Tokenizing is a choice, and every choice trades sequence length against vocabulary size. Three schemes; only one survives in practice.
Characters: tiny vocab, huge sequences
Split on every character and your vocabulary is tiny and you're never stuck on an unknown word — but sequence length explodes. Every token costs a forward-pass step, so huge sequences are slow.
Whole words: short sequences, giant vocab
Split on whole words and sequences get short, but the vocabulary becomes giant and you constantly hit out-of-vocabulary words the tokenizer has never seen.
Subwords (BPE): the middle path
Byte-level BPE keeps sequences short-enough, the vocabulary small, and — because it can always fall back to bytes — it is never stuck out-of-vocabulary. Rare words become several pieces; common words stay one. That's why every modern model uses it.
Keywords — tap to unfold the plain meaning
Characters = tiny vocab, huge sequences. Whole words = short sequences, giant vocab, OOV gaps. Subwords (BPE) = short-enough, small, never OOV. BPE wins.
How BPE actually merges
"Subword" sounds vague until you watch the algorithm build the pieces. It's just a greedy loop over the most common neighbors — run once at training, replayed forever.
Start from bytes, merge the top pair
Byte-Pair Encoding (BPE) starts from raw bytes, then repeatedly merges the most frequent adjacent pair into a new token, saving each merge as a rule. Run long enough, those saved rules are the vocabulary — about ~248k fixed pieces in Qwen3.6.
A toy worked example
Take the word "lowest". Merge the most frequent pair (s,t) → "st"; merge (l,o) → "lo"; merge (e,st) → "est". The word now tokenizes as lo · w · est — three tokens, each one a learned merge. Common pieces survive; rare ones stay split.
The BPE loop, decoded
- pieces ← bytesbegin with every text broken into raw bytes — never stuck, since bytes cover everything
- freq(pair)count how often each adjacent pair of pieces appears across the training text
- argmaxpairpick the single most frequent adjacent pair this round
- merge(·)fuse that pair into one new token and save the rule; repeat until the vocab is full
Greedy and simple: each round fuses the commonest neighbor pair into a new token. The ordered list of merge rules is the tokenizer; replaying it on new text reproduces the same pieces every time.
Keywords — tap to unfold the plain meaning
hello = 1 token, strawberry (with space) = 1 token (ID 70135), strawberry (no space) = 3 tokens, tokenization = 2 tokens, H100 = 4 tokens, every single digit = 1 token, 你好 = 1 token, and 🍓 = 3 byte pieces. Common strings collapse to one token; rare ones fragment.
BPE starts from bytes and greedily merges the most frequent adjacent pair, saving each rule. The saved rules are the ~248k-piece vocabulary.
The strawberry problem & chat-template overhead
Two consequences of tokenization bite in production: the model's blind spots about its own letters, and the silent tax the chat template adds to every single turn.
Why models miscount the r's in strawberry
Ask a model how many r's are in "strawberry" and it often gets it wrong. With a leading space the word is a single opaque token (integer 70135) — the model receives that number, not the letters s-t-r-a-w-b-e-r-r-y. So spelling, rhyming, and digit math are all hard: the letters simply aren't visible inside one token.
The chat template adds fixed overhead
Every turn is wrapped by a chat template with special tokens like <|im_start|> and <|im_end|>. "What is tokenization?" is 5 tokens raw, but 15 tokens once wrapped — a fixed tax on every request you can't skip.
It all lands in the token budget
Qwen3.6's context window is 131,072 tokens — the whole budget the prompt and generation share. In practice the prompt:generation ratio runs about 19–47:1: you pay for far more input tokens than you generate.
Tokens per string — measured on Qwen3.6, 4×H100, 2026-06-19
A leading space makes " strawberry" one opaque token; drop it and the same word fragments into three. Common = cheap, rare = fragmented.
Chat-template overhead, decoded
- rawthe 5 tokens of the user's actual text, "What is tokenization?"
- overheadthe fixed special tokens the chat template wraps around every turn (im_start, role, im_end…)
- wrapped = 15what the model actually counts — and what you're billed for — per turn
Short prompts pay the heaviest template tax in percentage terms: a 5-token question becomes 15 tokens, three times the size, before the model reads a word of it.
Keywords — tap to unfold the plain meaning
One opaque token hides its own letters, so spelling and digit math are hard. And the chat template turns a 5-token question into 15 — fixed overhead on every turn.
On your cluster
You don't have to guess token counts. The server tokenizes for you on request, and quietly tallies every token that crosses it.
Tokenize a string yourself
Post your text to the tokenize endpoint and the server returns the exact pieces and integer IDs — the same split prefill will use. This is how you check, before you're billed, how a string fragments.
Try it on the 4×H100 box
curl localhost:8000/tokenize -d '{ "model": "Qwen3.6", "prompt": "What is tokenization?" }'
"What is tokenization?" returns 5 tokens raw, 15 once the chat template wraps it. The vllm:prompt_tokens_total metric captures every prompt token the server processes — so input tokens, including all that fixed template overhead, show up in your bill and your dashboards whether you watch them or not.
Keywords — tap to unfold the plain meaning
Check yourself
- What is a token, in one sentence — and what does an integer ID actually index into?
- Why did subword (BPE) tokenization win over both character-level and whole-word schemes?
- Why is counting the r's in "strawberry" hard for the model, and roughly how many tokens does a short chat turn cost once the template wraps it?
The /tokenize endpoint shows your exact pieces and IDs, and vllm:prompt_tokens_total counts every token you pay for — template overhead included.