Glossary

The ubiquitous language for this course. Every lesson uses these terms exactly as defined here. Grows as we go.

Residual stream

The per-token vector path running through every transformer block. Attention and the MLP each compute an update that is added back to this stream. See Lesson 6.

MLP / SwiGLU

The feed-forward sublayer that transforms each token independently after attention. Its large projection matrices contain a major share of dense-model weights and decode memory traffic.

Mixture of Experts (MoE)

A model with many expert MLPs where a router selects a small subset for each token. Total parameters drive storage; active parameters better approximate expert compute; routing adds load balance and communication costs. See Lesson 7.

Reproducible benchmark

A comparison with fixed model/engine versions, hardware, precision, input/output distributions, load model, sampling, cache state, SLO, and quality target. Reports latency distributions and goodput, not an isolated peak tokens/sec. See Lesson 16.

Token

The atomic unit of model input and output: a piece of text (usually a subword) paired with an integer ID, its index in the model's fixed vocabulary. The model only ever sees these integers, never your letters. Averages ≈ ¾ of an English word (~4 characters). See Lesson 3.

Tokenizer

The component that converts text ↔ token IDs (encode and decode), using a learned vocabulary plus BPE merge rules. Fixed per model and loaded by the server (vLLM exposes it at /tokenize).

Vocabulary

The fixed set of all tokens a model knows (your Qwen3.6 = 248,320). A token's position in this set is its integer ID.

Subword

A token that is a fragment of a word. Frequent words are a single subword; rare words split into several, so any string is representable and nothing is "out-of-vocabulary" (it can fall back to raw bytes).

Byte-Pair Encoding (BPE)

The algorithm that builds the vocabulary: starting from raw bytes, repeatedly merge the most frequent adjacent pair into a new token, saving each rule. Byte-level BPE operates on UTF-8 bytes, so a leading space becomes part of the next piece (shown as Ġ) and any character is always encodable.

Special tokens

Reserved tokens that aren't ordinary text: end-of-sequence (EOS), and chat-role markers like <|im_start|> / <|im_end|>. Added around your content before prefill.

Chat template

The model-specific pattern that wraps your messages in special tokens (system / user / assistant roles) before tokenization. It is what actually gets prefilled: fixed overhead on every turn (Qwen3.6 even auto-opens a <think> block). See Lesson 3.

Context window

The maximum number of tokens (prompt + generation, all held in the KV cache) a model can attend to at once; your Qwen3.6 serves 131,072. Exceeding it forces truncation or eviction.

Prefill

The first phase of inference: the model processes the input prompt to populate the KV cache and produce the first output token. Engines may process a long prompt in chunks rather than one monolithic pass. Prefill is usually compute-bound for sufficiently large prompts and efficient kernels; short prompts and underfilled hardware can behave differently.

Decode

The second phase: the model generates output tokens one at a time, each step consuming the previously generated token. Each step does little arithmetic but streams weights and reads the growing KV cache. It is usually memory-bandwidth-bound at modest batch sizes; larger batches improve weight reuse and can move some layers toward compute-bound.

KV cache

The stored key and value vectors for every past token, kept so attention doesn't recompute them each decode step. It grows linearly with sequence length and active sequences. Whether it dominates memory depends on model size, context length, concurrency, KV precision, and the memory reserved for weights and runtime workspaces.

Memory-bandwidth-bound

A workload limited by how fast data moves from GPU memory (HBM), not by how fast the GPU can compute. Speeding up the math doesn't help; you must move less data or move it faster. Decode commonly lives here at modest batch sizes.

Compute-bound

A workload limited by the GPU's arithmetic throughput (FLOPs / tensor cores), not by memory traffic. Long-prompt prefill commonly lives here. Decode's matrix multiplications can approach this regime as batch grows, but there is no universal crossover batch: it depends on hardware, precision, model shape, kernels, and how much attention/KV work remains.

Arithmetic intensity

FLOPs performed per byte read from memory. High intensity → compute-bound; low intensity → memory-bound. Prefill generally has higher intensity than single-request decode. The actual value depends on prompt length, batch, precision, model shape, and implementation.

Roofline

A plot of achievable throughput vs arithmetic intensity: a rising bandwidth-limited slope that flattens into a compute-limited roof. Tells you, for any workload, whether speeding up compute or memory will help. See Lesson 17.

Ridge point

Where the roofline's slope meets its roof: ridge = peak FLOPs ÷ memory bandwidth (FLOP/byte). Intensity below it → memory-bound; above → compute-bound. H100 NVL ≈ 214 FLOP/byte (BF16 dense).

Tensor parallelism (TP)

Split each layer's weight matrices across N GPUs so one model runs on several GPUs at once. Needs communication collectives—commonly all-reduce, reduce-scatter, or all-gather—to recombine or redistribute partial results through each layer. Use when a model won't fit on one GPU, or for lower single-stream latency over NVLink. See Lesson 27.

All-reduce

A collective that sums a tensor across all participating GPUs and returns the result to each. Classic Megatron-style tensor parallelism performs two synchronization collectives per transformer block; newer implementations may use reduce-scatter/all-gather variants. During decode these collectives are on each token's critical path.

NVLink

NVIDIA's high-bandwidth GPU-to-GPU interconnect (H100 ≈ 900 GB/s), roughly 7× faster than PCIe (≈ 128 GB/s). Decisive for TP, whose all-reduce rides it. In the analogy: the express lane between prep stations.

Latency vs. throughput

Latency = time for one request (e.g. time-to-first-token, then per-token). Throughput = tokens/sec across all requests. Batching trades latency for throughput; the two phases sit on opposite sides of this trade. The knee is where throughput saturates but latency keeps climbing, so run just below it.

TTFT (time to first token)

Request arrival → first output token, dominated by prefill. The first SLO users feel, protected by chunked prefill + prefix caching.

TPOT / ITL (time per output token)

Average gap between output tokens once generation starts, dominated by decode. The streaming-smoothness SLO; per-request throughput ≈ 1 ÷ TPOT.

Goodput

The throughput that meets both the TTFT and TPOT SLOs. Past the knee, raw throughput can stay flat while goodput collapses, so optimize for goodput, not throughput. See Lesson 15.

Little's Law

in-flight requests = throughput × latency. Turns a latency target + the concurrency cap (max-num-seqs) into max QPS and replica count: the bridge from the knee to autoscaling.

HBM (high-bandwidth memory)

The GPU's main memory, where model weights and the KV cache live. Big but "far": reaching it is the cost that makes decode memory-bound. In the analogy: the pantry across town.

Attention head

One of several parallel "tasters" inside attention. Per token, each head emits a Query (what it's looking for), a Key (how it labels itself), and a Value (what it carries), each a vector of head_dim numbers. The KV cache stores the K and V of each KV head.

head_dim

The length of one head's Key (or Value) vector: how many numbers describe a token from that head's angle (128 in Llama 2 7B). In the formula, kv_heads × head_dim is the width of one token's cache entry per layer (the index card's length).

GQA (grouped-query attention)

An attention variant where several query heads share one key/value head, so the model stores fewer KV heads. Shrinks the KV cache by the group factor (e.g. Llama 3 8B: 32 query ÷ 8 KV heads = 4× smaller). The middle ground between full multi-head attention (MHA) and single-KV MQA.

Quantization

Store weights, activations, and/or the KV cache in fewer bits (FP16 → FP8 → INT4), rounding each number onto a coarser grid. Shrinks memory and the bytes moved per token (faster decode), at a small accuracy cost. The risk lives in a few outlier values. See Lesson 24.

FP8 (E4M3 / E5M2)

8-bit floating point. E4M3 (4 exponent, 3 mantissa bits: higher precision, small range) is used for weights/activations/KV; E5M2 offers wider range. H100 tensor cores have higher peak FP8 throughput than BF16, but quality is not automatically lossless: it depends on calibration/scaling, which tensors are quantized, the model, and the evaluation workload.

Static batching

Group N requests, run them together, and wait for all to finish before starting the next batch. When request lengths vary, finished slots idle until the slowest ends. The waste is workload-dependent; continuous batching removes finished requests at iteration boundaries instead of assuming one universal idle percentage.

Continuous batching (iteration-level scheduling)

Schedule one decode step at a time: after each step, evict finished requests and admit waiting ones, so the running batch never drains. Introduced by Orca; keeps the memory-bound decode phase fed. See Lesson 18.

PagedAttention

Store the KV cache in small fixed-size blocks allocated on demand and noncontiguously: virtual-memory-style paging for the GPU. It sharply reduces reservation and fragmentation waste compared with contiguous per-request allocation; the realized capacity gain depends on block size and workload. vLLM's original paper reported substantial waste in prior systems, not a universal saving for every deployment. Lesson 18.

Prefix caching

Reuse PagedAttention blocks across requests that share a prompt prefix (e.g. a common system prompt) instead of recomputing/restoring their KV, a big win for RAG.

Prompt caching

An API feature that lets repeated stable prefixes avoid some prefill work. Prefix-KV reuse is a common implementation, but provider internals are not guaranteed by the product name. TTLs, eligibility rules, and read/write prices are provider-specific and change over time; consult the provider's current documentation. See Lesson 20.

Cache TTL

How long a cached prefix survives before eviction. Anthropic's default is 5 minutes (refreshed on each use), with a 1-hour option. Idle past the TTL and the next request is a cache miss that re-prefills from scratch.

Tensor

A multi-dimensional array of numbers (a generalization of a vector or matrix): the basic data object that flows through a model. Both the weights and the activations are tensors. See Lesson 2.

Embedding

The learned vector a token ID maps to, carrying the token's meaning (and position) into the first layer. The embedding table is the big lookup from every vocabulary ID to its vector. See Lesson 4.

Dot product

Multiply two vectors' paired numbers and add the results into one score: larger when the vectors point the same way. It is how attention scores one token's Query against another token's Key. See Lesson 5.

Projection (learned)

Multiplying a vector by a learned matrix to map it into a new vector: the basic operation that turns a token's vector into its Query, Key, and Value, and that the final LM head uses to score the vocabulary. "Learned" means the matrix's numbers are part of the model's weights. See Lesson 5.

Softmax

A function that squashes a list of raw scores into positive weights that add up to 1: it turns attention scores into mixing weights, and final logits into next-token probabilities. See Lesson 5.

Normalization (LayerNorm / RMSNorm)

A step that rescales each token's vector to a controlled magnitude so a deep stack of layers stays numerically stable. RMSNorm is a cheaper variant; pre-norm applies it before each sublayer (attention and MLP). See Lesson 6.

Hidden size

How many numbers are in each token's vector as it flows through the model: the width of the residual stream. Larger hidden size means more parameters per layer. See Lesson 6.

Activation

An intermediate vector a layer produces as data flows through the model (as opposed to the fixed weights). It is the A in quantization notation like W8A8; its rare large "outlier" values are what make activations hard to quantize. See Lesson 6.

Logits

The raw, unnormalized scores the model emits for every token in the vocabulary at each step, before softmax turns them into probabilities. See Lesson 8.

FLOP / TFLOPS

A FLOP is one floating-point math operation (a single multiply or add); a model's compute cost is counted in FLOPs. TFLOPS = trillions of FLOPs per second, a chip's compute-rate ceiling (e.g. H100 ≈ 836 TFLOPS dense BF16). See Lesson 17.

Tensor Core

The GPU's dedicated matrix-multiply units: the hardware that supplies a GPU's headline FLOPs. When a workload is compute-bound, these are what saturate. See Lesson 29.

BF16 / FP16

16-bit floating-point formats, 2 bytes per number: the common "full precision" for serving, versus 1-byte FP8. BF16 trades mantissa bits for a wider exponent range than FP16. See Lesson 24.

W8A8 / W4A16 (weight/activation bits)

Shorthand for how many bits a quantized model uses: W = weight bits, A = activation bits. W8A8 = 8-bit weights and 8-bit activations; W4A16 = 4-bit weights with 16-bit activations (weight-only). See Lesson 24.

Chunked prefill

Splitting a long prompt's prefill into smaller pieces so it can be interleaved with ongoing decode in the same batch, instead of stalling everyone's decode while one big prompt is processed. See Lesson 18.

RAG (retrieval-augmented generation)

Fetching documents at request time and pasting them into the prompt so the model can answer from them. It makes prompts long and largely shared across requests, which is why RAG workloads are prefill-heavy and benefit so much from prefix caching. See Lesson 20.

Rejection sampling (speculative decoding)

The accept/reject rule that makes speculative decoding lossless: accept each drafted token with a probability that corrects the draft model's bias, and resample any rejected token from a corrected distribution, so the output matches the target model's own distribution exactly. See Lesson 21.

Perplexity

A generic measure of how well a model predicts held-out text (lower is better). Useful as a regression signal but a poor stand-in for task quality, so it is rarely the product metric. See Lesson 22.

Computation graph

A model expressed as a graph of math operations (MatMul, Softmax, …) on tensors, not just its weights: the form that lets a runtime like ONNX or TensorRT replay or compile the steps. See Lesson 26.