⚙ Inference Engineering 0 / 36

A calm course on serving large language models

Inference Engineering, explained calmly

36 short, scannable lessons that take you from "what is an LLM" all the way to running one on a real 4×H100 cluster — the runtime, the hardware, and production serving. Real terms, real math, and real measured numbers, never dumbed down.

The idea You're a lightning-fast line cook. The weights live in a pantry across town, reached by one road with one van. Compute is your hands; HBM is the pantry; memory bandwidth is that road. Almost everything in serving is about that one road.
Start the course

Part I How an LLM Works

The model itself — from a next-token guess to a full forward pass.

  1. 01What Is an LLM?Training vs inference, next-token prediction, and the three-stage request pipeline.
  2. 02What's Inside a ModelModels as numbered tensor arrays stored in SafeTensors.
  3. 03TokenizationTurning text into integer tokens — the unit you actually pay for.
  4. 04EmbeddingsToken IDs become learned vectors that carry meaning and position.
  5. 05AttentionQ/K/V, softmax, causal masking, and multi-head attention.
  6. 06Inside a Transformer BlockNorms, the residual stream, attention, MLP/SwiGLU, and the LM head.
  7. 07Architecture VariantsDense vs mixture-of-experts; MHA, GQA and MQA in serving.
  8. 08The Forward Pass & SamplingBlocks → logits → a token, via greedy, temperature and top-k.
  9. 09The Autoregressive Loop & CostToken-by-token generation and where the cost quietly escalates.
  10. 10One Request, End to EndHTTP → chat template → tokenize → queue → stream back.
  11. 11Practical API ControlsSampling limits, streaming, cancellation, logprobs, structured output.
TransferDraw the token→token loop: mark where tokenization, embeddings, attention, logits and sampling sit, then circle the work that repeats every single token.

Part II The Inference Runtime

How a real engine makes that loop fast — and the tradeoffs it juggles.

  1. 12What Is an Inference Engine?The naive loop vs vLLM, SGLang, TensorRT and Dynamo.
  2. 13Prefill vs DecodeThe asymmetry behind every latency and throughput tradeoff.
  3. 14The KV Cache & Its MemorySizing the cache and understanding what caps your batch.
  4. 15Latency, Throughput & SLOsTTFT, TPOT, the knee, goodput, and Little's Law.
  5. 16Benchmarking CorrectlyWorkload shape, warmup, percentiles and reproducibility.
  6. 17The RooflineOps-per-byte and when decode is memory- vs compute-bound.
  7. 18PagedAttention & BatchingWhy naive batching wastes cache; keeping decode full.
  8. 19CUDA Kernels & FusionCutting HBM round-trips with FlashAttention and SRAM tiling.
  9. 20Prefix Caching & the KV HierarchyReusing KV across requests; the memory hierarchy and routing.
  10. 21Speculative DecodingA draft model proposes tokens; the target verifies in parallel.
  11. 22Quality EvaluationQuality gates for quantization, engines, sampling and rollouts.
  12. 23Failure Modes & DebuggingOOM, context, queues, templates, compatibility, lifecycle.
TransferA service has good TTFT but poor TPOT and low batch occupancy: pick three measurements, predict the bottleneck, and rank batching, quantization, prefix caching and speculative decoding by likely impact.

Part III Precision & Formats

Spend fewer bits per number — without quietly wrecking quality.

  1. 24Quantization: Number FormatsFP8, INT8 and INT4 — what each shrinks and costs.
  2. 25Quantization AlgorithmsGPTQ, AWQ and SmoothQuant, and the activation-outlier problem.
  3. 26Model Formats & CompilationSafeTensors, ONNX and TensorRT: serialization vs compilation.
TransferCompare BF16, FP8 and weight-only INT4 for a workload: what shrinks, what may speed up, what needs hardware support, and what quality evidence you'd want.

Part IV Scaling Across GPUs

When one GPU isn't enough — splitting the model and the work.

  1. 27Model Parallelism & NVLinkSplitting a model across GPUs and the role of the interconnect.
  2. 28Disaggregated ServingSeparate prefill and decode onto distinct workers (Dynamo).
TransferA model fits on one GPU but misses latency targets: choose replicas, tensor parallelism, or disaggregation — and name the communication cost that could change your mind.

Part V The Hardware

The metal under it all — SMs, HBM, generations, and slicing a GPU.

  1. 29GPU Architecture: SMs & HBMStreaming multiprocessors, Tensor Cores, and the HBM/SRAM hierarchy.
  2. 30GPU GenerationsHopper, Blackwell and Rubin: precision, interconnect, capacity.
  3. 31Multi-Instance GPU (MIG)Slicing one GPU into right-sized, multi-tenant instances.
TransferPlace one compute-heavy prefill and two latency-sensitive decode workloads on your GPUs: defend your generation, interconnect and MIG/time-slicing choices.

Part VI Production Serving

Run it for real — routing, scaling, shipping, and cost.

  1. 32Routing, Load Balancing & QueueingToken- and KV-aware routing; queue behavior under load.
  2. 33AutoscalingConcurrency targets, scaling signals, and cold starts.
  3. 34Containerization: Docker & NIMReproducible serving containers via Docker and NVIDIA NIM.
  4. 35Multi-Cloud CapacityInference fleets across clouds: supply, latency, reliability.
  5. 36Zero-Downtime Deployment & CostBlue-green and canary with rollback; $/1M-token costing.
CapstoneGiven a workload mix and SLOs, propose routing, concurrency, autoscaling, rollout and cost. Report goodput and $/1M tokens, and name the measurement most likely to break your plan.