John Lam · Inference Engineering my learning journal

Inference Engineering

Bridging from the ops layer down into the GPU internals, taught against a real 4×H100 cluster. Start with Lesson 1, or open the Lab.

The running analogy: compute is a line cook; the weights live in a pantry across town; the one road between them is memory bandwidth.

Choose your path

Already operate inference systems? Start from a symptom.

The numbered course builds from model fundamentals upward. The operator path starts from familiar production pain, then traces downward until the cause is explainable from first principles:

  1. Define the symptom and SLO: TTFT, TPOT, goodput, and the knee.
  2. Separate prefill from decode, then use KV memory, roofline, and batching to explain the bottleneck.
  3. Check the hardware boundary: fit, interconnect, and replica-versus-TP choices.
  4. Return to operations: route, scale, deploy, and cost the system using the internals you just recovered.

New to LLM internals? Follow Lessons 1–36 in order; both paths cover the same material.

Studied a few? Lock them in.

Spaced Review → resurfaces the recall questions from across every lesson on a spacing schedule, interleaved across topics. Retrieval practice is what turns "I read it" into "I remember it." (Press / on any lesson to jump here too.)

Your progress

A quick journal of where you are: opened lessons fill in, and they highlight once you have cards in spaced review.

Lessons

Part I · How an LLM works

Lesson 1
What Is an LLM?
Training vs inference, next-token prediction, the pipeline.
Lesson 2
What's Inside a Model
A model is just numbered tensor arrays: SafeTensors.
Lesson 3
Tokenization
Text → integer tokens, the unit everything is measured in.
Lesson 4
Embeddings
Token IDs → learned vectors that carry meaning + position.
Lesson 5
Attention
Q/K/V, softmax, causal mask, multi-head: tokens read each other.
Lesson 6
Inside a Transformer Block
Norms, residual stream, attention, MLP/SwiGLU, and LM head.
Lesson 7
Architecture Variants
Dense vs MoE; MHA vs GQA vs MQA—and why serving changes.
Lesson 8
The Forward Pass & Sampling
Blocks → logits → token: greedy, temperature, top-k.
Lesson 9
The Autoregressive Loop & Cost
The naive loop, and why each step gets slower (the staircase).
Lesson 10
One Request, End to End
HTTP → template → tokens → queue → prefill → stream → stop.
Lesson 11
Practical API Controls
Limits, sampling, streaming, cancellation, logprobs, structured output.
Part I transfer: Given a chat request, draw the token-to-token loop and mark exactly where tokenization, embeddings, attention, logits, and sampling occur. Then identify which work repeats for every generated token.

Part II · The inference runtime

Lesson 12
What Is an Inference Engine?
The naive loop vs vLLM/SGLang/TRT/Dynamo, and what they add.
Lesson 13
Prefill vs Decode
The asymmetry behind every latency/throughput tradeoff.
Lesson 14
The KV Cache & Its Memory
Compute the cache; see why it caps your batch.
Lesson 15
Latency, Throughput & SLOs
TTFT, TPOT, the knee, goodput, and Little's Law.
Lesson 16
Benchmarking Correctly
Workload shape, warmup, load model, percentiles, and reproducibility.
Lesson 17
The Roofline
ops:byte, and when decode is memory- vs compute-bound.
Lesson 18
PagedAttention & Batching
Why naïve batching wastes the cache; keeping decode fed.
Lesson 19
CUDA Kernels & Fusion
Cutting HBM round-trips; FlashAttention and SRAM tiling.
Lesson 20
Prefix Caching & the KV Hierarchy
Reusing KV across requests; the memory hierarchy; routing.
Lesson 21
Speculative Decoding
A draft model proposes; the target verifies in parallel.
Lesson 22
Quality Evaluation
Quality gates for quantization, engines, sampling, and rollouts.
Lesson 23
Failure Modes & Debugging
OOM, context, queues, templates, compatibility, and request lifetime.
Part II transfer: A service has good TTFT but poor TPOT and low batch occupancy. Choose three measurements, predict the bottleneck, and rank batching, quantization, prefix caching, and speculative decoding by likely impact.

Part III · Precision & formats

Lesson 24
Quantization: Number Formats
Fewer bits per number: FP8/INT8/INT4 on your cluster.
Lesson 25
Quantization Algorithms
GPTQ, AWQ, SmoothQuant: taming activation outliers.
Lesson 26
Model Formats & Compilation
SafeTensors, ONNX, TensorRT: serialize vs compile.
Part III transfer: Compare BF16, FP8, and weight-only INT4 for one workload. State what shrinks, what may speed up, what requires hardware/kernel support, and what quality evidence you would demand.

Part IV · Scaling across GPUs

Lesson 27
Model Parallelism & NVLink
Splitting a model across GPUs, and the wire that decides it.
Lesson 28
Disaggregated Serving
Split prefill & decode onto separate workers (Dynamo).
Part IV transfer: A model fits on one GPU but misses its single-stream latency target. Decide whether to use replicas, TP, or prefill/decode disaggregation, and name the communication cost that could reverse your choice.

Part V · The hardware

Lesson 29
GPU Architecture: SMs & HBM
SMs, Tensor Cores, and the HBM/SRAM memory hierarchy.
Lesson 30
GPU Generations
Hopper → Blackwell → Rubin: precision, interconnect, capacity.
Lesson 31
Multi-Instance GPU (MIG)
Right-size instances; slice one GPU for multi-tenant serving.
Part V transfer: Place one compute-heavy prefill workload and two latency-sensitive decode workloads on the available GPUs. Defend the generation, interconnect, and MIG/time-slicing choices.

Part VI · Production serving

Lesson 32
Routing, Load Balancing & Queueing
Token-aware & KV-aware routing; queues under load.
Lesson 33
Autoscaling
Concurrency targets, scaling signals, and cold starts.
Lesson 34
Containerization: Docker & NIM
Reproducible inference containers, via Docker and NVIDIA NIM.
Lesson 35
Multi-Cloud Capacity
Inference fleets across clouds: supply, latency, reliability.
Lesson 36
Zero-Downtime Deployment & Cost
Blue-green/canary deploys with rollback; $/1M-token cost.
Part VI capstone: Given a workload distribution and TTFT/TPOT SLO, propose routing, concurrency, autoscaling, rollout, and cost assumptions. Report goodput and $/1M tokens, then name the measurement most likely to invalidate your plan.

Reference

Lab
Your 4×H100 Cluster
A live cluster, dated live-tested results, and the vLLM-flag → lesson map.
Ops · action needed
Cluster Findings
NVLink pair down, imbalance, tuning proposal for the 4×H100.
Analogy
The Faraway Pantry
The running metaphor every lesson reuses.
Reference
Glossary
The course's shared vocabulary.
Why
Mission
The goal grounding every lesson.
Sources
Resources
Curated, high-trust reading.
Refresh live cluster numbers anytime: bash learning/courses/inference-engineering/tools/cluster-probe.sh