Resources

Curated, high-trust sources. Lesson knowledge is drawn from here.

Knowledge

• Transformer, RMSNorm, and GLU variants: the complete block (Lesson 6).
• Grouped-Query Attention and Switch Transformers: inference-relevant architecture variants (Lesson 7).
• vLLM serving API, MLPerf Inference, LM Evaluation Harness, and vLLM troubleshooting: request controls, benchmarking, quality, and diagnosis (Lessons 10–11, 16, 22–23).
• LLM Inference Handbook (BentoML): canonical mental models & vocabulary (prefill/decode, KV cache, batching, disaggregation).
• Mastering LLM Techniques: Inference Optimization (NVIDIA): hardware-grounded overview.
• Transformer Inference Arithmetic (kipply): THE rigorous arithmetic source (KV cache, weights, intensity).
• How to Scale Your Model: Inference (JAX ML): deep roofline/throughput treatment.
• vLLM: PagedAttention & Continuous Batching (RunPod): how a real engine solves KV waste.
• Prefill Compute-Bound, Decode Memory-Bound (TDS): the phase-asymmetry roofline intuition.
• SARATHI (arXiv 2308.16369): primary paper behind chunked prefill.
• PagedAttention / vLLM (Kwon et al., SOSP'23): primary paper for paging the KV cache (Lesson 18).
• Orca (Yu et al., OSDI'22): primary paper for continuous batching / iteration-level scheduling (Lesson 18).
• Roofline (Williams et al., CACM 2009): the original roofline model (Lesson 17); see also the Modal GPU Glossary.
• Prompt caching (Anthropic docs): read/write multipliers + TTL; backs the Lesson 18 prompt-caching bridge.
• FP8 Formats (Micikevicius et al.), AWQ, SmoothQuant, and the vLLM FP8 KV-cache blog: the quantization sources for Lesson 24.
• Megatron-LM (Shoeybi et al., 2019): primary paper for tensor parallelism (Lesson 27); plus NVLink vs PCIe.
• Sarathi-Serve (Agrawal et al., OSDI 2024): the latency-throughput knee / chunked prefill (Lesson 15); plus DistServe (goodput) and Little's Law.

Production serving

• Dynamo routing concepts (NVIDIA): official KV-overlap and decode-load routing cost model (Lesson 32).
• Horizontal Pod Autoscaling (Kubernetes) and KEDA scaling: official custom-metric and event-driven scaling behavior (Lesson 33).
• Pod Lifecycle (Kubernetes): authoritative termination flow, preStop, signals, and grace periods (Lesson 36).
• NVIDIA Container Toolkit and NIM on Kubernetes: authoritative GPU-container and NIM deployment references (Lesson 34).
• Kubernetes cluster administration: official multi-cluster boundary for hybrid and multi-cloud designs (Lesson 35).
• vLLM production metrics: queue, running-request, cache, TTFT, and inter-token-latency semantics (Lessons 15, 32, and 33; verify against the deployed version).

Wisdom (Communities)

• r/LocalLLaMA: practitioner serving tradeoffs, hardware sizing, framework comparisons.
• vLLM GitHub, where serving-engine details get argued in the open.

Community participation is optional, listed for testing understanding against practitioners.

Gaps

• Rigorous arithmetic source: closed (kipply + JAX scaling book above).
• PagedAttention / KV-fragmentation source for Lesson 18: closed (vLLM + Orca papers above).