A live 4×H100 OpenShift cluster running vLLM models: the course's measuring instrument. The course is tested against this cluster; stored results retain their test date, and the probe refreshes current operational values.
The calculator gives you a prediction; this cluster lets you
test it. On 2026-06-19, one Qwen3.5-27B run measured 3,288 prompt tok/s vs
~108 generation tok/s. That is evidence for this configuration and workload,
not a universal model constant. Refresh with bash learning/courses/inference-engineering/tools/cluster-probe.sh
before drawing a current conclusion.
Verified 2026-06-19: only GPU2↔GPU3 has working NVLink (12 links). GPU0↔GPU1 report zero links. A 4×H100 NVL box should have two bridged pairs, but you currently have one, and the tensor-parallel model is sitting on the broken pair. Full diagnosis, severity ranking + a rebalance proposal → Cluster Findings & Tuning Proposal.
| GPUs | 4 × NVIDIA H100 NVL, 94 GiB HBM3 each (95,830 MiB) |
|---|---|
| Logical GPUs | 20 (each physical GPU time-sliced ×5, MIG disabled) |
| Node | 128 vCPU · ~2 TiB RAM · single worker (control-plane + worker) |
| Power/thermal | 400 W cap/GPU; idle ~60–110 W, 38–64 °C when not generating |
| Namespace | Model | GPUs | Notable serving flags |
|---|---|---|---|
| llm-serving-qwen36 | Qwen3.6-27B-FP8 (vLLM) | TP 1 | util 0.75 · 128K ctx · max-seqs 8 · kv fp8 · chunked prefill · prefix cache |
| qwen35-27b-fixed | Qwen3.5-27B-FP8 (vLLM) | TP 2 | util 0.92 · 128K ctx · max-seqs 64 · kv fp8 · chunked prefill |
| llm-serving-embedding | BGE-M3 embedding | 1 | retrieval encoder |
| llm-serving-reranker | BGE-M3 reranker | 1 | retrieval reranker |
| bert-ner | BERT NER | 1 | entity extraction |
Also present but not on GPU right now:
llama, gemma4, gpt-oss, qwen3-vl,
classifier, sentiment, bm25 namespaces (scaled to 0).
This is the bridge from your ops layer to the internals. Each serving flag you set is one of the levers these lessons explain:
| vLLM flag (on your cluster) | What it controls, and the lesson |
|---|---|
| --kv-cache-dtype fp8 | 1 byte/KV element → halves the cache vs FP16. Lesson 14 |
| GQA: 4 KV heads (arch) | fewer kv_heads → 6× smaller cache for Qwen3. Lesson 14 |
| --gpu-memory-utilization | fraction of 94 GiB vLLM claims for weights + KV. Lesson 14 |
| --max-model-len 131072 | per-request context ceiling; KV grows linearly with it. Lesson 14 |
| --max-num-seqs | max concurrent sequences: a scheduler and memory cap. Lesson 14 and Lesson 18 |
| --max-num-batched-tokens | token budget per engine step (prefill+decode). Lesson 13 and Lesson 18 |
| --enable-chunked-prefill | split long prefills and schedule them alongside decode. Lesson 18 |
| --enable-prefix-caching | reuse KV blocks for shared prompt prefixes. Lesson 20 |
| --tensor-parallel-size 2 | split one model across 2 GPUs; interconnect affects collective cost. Lesson 27 |
bash learning/courses/inference-engineering/tools/cluster-probe.sh and record those
fields alongside the raw output.
The course probe completed against the current OpenShift context as
kube:admin, reading all four H100 NVLs and both vLLM deployments. Run
bash learning/courses/inference-engineering/tools/cluster-probe.sh to replace this verification with the current state.
| Prefill vs decode | Qwen3.5-27B: prompt ~3,288 tok/s vs generation ~108 tok/s (single request, measured 2026-06-19). Re-run before comparing another engine or model. |
|---|---|
| Workload shape | prompt:generation ratio 19:1 (qwen36) and 47:1 (qwen35): heavily prefill-bound RAG |
| KV cache usage | 0.2–0.5% under light load (short RAG contexts vs the 128K ceiling) |
| Refresh | bash learning/courses/inference-engineering/tools/cluster-probe.sh |