Containerization: Docker & NIM — Inference Engineering

01

The inference container

TL;DR · A serving image is just CUDA base + inference engine + model config — the same containerization you already do, with three GPU-shaped twists.

You already ship services in containers. An LLM server is one more container — but the pod needs a GPU, the readiness check has to wait for weights, and the image is enormous. Get those three and the rest is your normal pipeline.

What goes in the image

A serving image combines an nvidia/cuda base + an inference engine (vLLM) + the model configuration. The NVIDIA Container Toolkit exposes GPU access to the container; the engine serves an OpenAI-compatible API and exports Prometheus /metrics.

Three things that make it special

Versus any other service, the differences are exactly three: the pod needs a GPU, the readiness probe must wait for tens of GB of weights to load, and the images are huge — so layer caching and weight placement matter.

Today's win

By the end you should be able to predict the GPU-specific containerization aspects and decide when NIM beats a custom Dockerfile. Predict first, then reveal — even a wrong guess makes it stick.

Keywords — tap to unfold the plain meaning

Build up from CUDA, add the engine, add the model — then serve an OpenAI-shaped API and a Prometheus metrics endpoint. The Container Toolkit is what lets the container reach the GPU.

An inference container is CUDA base + engine + model. It's like every other service except for the GPU, the probe, and the image size.

02

The pod needs a GPU

TL;DR · A normal pod can't see the GPU. You request nvidia.com/gpu in the pod spec and the NVIDIA Container Toolkit wires the device through.

Containers are walled gardens by design — they don't get the host's GPU unless you ask. Two pieces make it work, and forgetting either means a server that boots and then can't find any hardware to run on.

Request the device

In the pod spec you request nvidia.com/gpu as a resource — the same way you'd request CPU or memory. The Kubernetes scheduler then only places the pod on a node that actually has a free GPU to give it.

Wire it through

The NVIDIA Container Toolkit is what actually exposes the host GPU and drivers inside the container. Request the resource and have the toolkit installed — both, or the container starts blind to the hardware.

Keywords — tap to unfold the plain meaning

Analogy ❝The container is a line cook locked in a prep room with no stove. Requesting nvidia.com/gpu is putting in the order for a station with a stove; the Container Toolkit is the door that actually connects that prep room to the burners. Order a stove but leave the door bricked up and the cook still can't cook — you need both the request and the open door.

GPU access = request nvidia.com/gpu in the pod spec plus the NVIDIA Container Toolkit to wire the device through. Both, or the pod runs blind.

03

Probes must wait for weights

TL;DR · A normal readiness probe passes at process start. An LLM isn't ready until tens of GB are loaded into VRAM — gate readiness on that, or you route traffic to a dead pod.

This is the one that bites people. The process is up in a second; the model is usable a minute later. If the load balancer believes the process, it sends real requests to a pod that's still loading — and every one fails.

The critical twist

Standard readiness probes pass the moment the process starts. But an LLM server needs tens of GB of weights loaded into VRAM before it can answer. If readiness passes prematurely, load balancers route traffic to unresponsive pods.

Gate on engine-ready, not process-up

Make readiness reflect the actual engine-ready state — only green once weights are loaded and the server can generate. Pair it with a generous startupProbe / start-period so the platform doesn't kill the pod mid-load.

The correct sequence

The order is: start process → load weights into VRAM → READY → take traffic. Readiness must not pass during startup; only after weight loading completes does the pod join the rotation.

Keywords — tap to unfold the plain meaning

The process is up almost instantly, but the pod is only truly READY after the weights finish loading into VRAM. Gate readiness on that last green box — never on "process started".

SRE note On the cluster, the vLLM deployment serves ~27 GB FP8 weights. Readiness gating is critical here — incorrect probe timing causes rollouts to route traffic to cold pods. Use a generous startupProbe so the platform tolerates a minute-long load instead of killing the pod and looping.

Process-up ≠ ready. Gate readiness on weights loaded into VRAM, with a generous startupProbe — otherwise the load balancer sends traffic to a dead pod.

04

The images are huge

TL;DR · Multi-GB weights make these images enormous, so you care about layer caching, registry locality, and whether weights are baked into the image or mounted at runtime.

A web service image is a few hundred MB. An LLM image carries the engine and possibly tens of GB of weights. At that size, "just docker pull it" becomes a real cost — pull time, registry bandwidth, cold-start latency.

Layer caching & registry locality

Order your layers so the giant, rarely-changing parts (CUDA, engine) are cached and reused across builds and pulls. Keep a registry close to the cluster — registry locality cuts pull time when nodes are fetching tens of GB.

Baked vs mounted weights

The big decision: baked-in weights (in the image — self-contained, but a massive image) vs mounted weights (pulled from a volume or object store at runtime — small image, but a load step at startup). This is a real trade-off, not a default.

Keywords — tap to unfold the plain meaning

Analogy ❝Weights are a giant pantry. Baking them into the image is building the pantry into the kitchen — the truck that delivers the kitchen is huge and slow, but the cook starts the instant it arrives. Mounting them is keeping the pantry in a faraway warehouse — the kitchen ships light and fast, but the cook waits while every ingredient is carted in before service. Keeping that warehouse near the kitchen — registry locality — is what keeps the wait short.

Huge images mean three levers: layer caching, registry locality, and baked vs mounted weights. Trade fat-but-instant against thin-but-loads-at-startup.

05

NIM vs custom Dockerfile

TL;DR · NIM is a prebuilt, optimized container — the fast path with little tuning. A custom Dockerfile gives full control over versions, flags, and patches. It's convenience vs flexibility.

Two ways to get to a running server. One hands you a tuned image and a standard API; the other hands you a blank CUDA base and total control. Knowing which to reach for is the whole point of this lesson.

NVIDIA NIM — the fast path

NVIDIA NIM ships prebuilt, optimized inference containers — often with TensorRT-LLM engines — behind a standard API. You get near-peak performance with little tuning required. Less control, but you're serving in minutes.

Custom Dockerfile — full control

A custom Dockerfile (CUDA base + vLLM) gives full control over engine versions, runtime flags, and patches. More work and more tuning, but you can pin exactly what you need and apply fixes NIM hasn't shipped.

The trade-off, named

It's convenience and peak-performance (NIM) versus flexibility (custom). NIM for a fast, optimized standard path; a Dockerfile when you must control versions and behavior.

Keywords — tap to unfold the plain meaning

Analogy ❝NIM is a meal kit from a great restaurant — portioned, pre-tuned, on the table fast, and you don't pick the brand of every ingredient. A custom Dockerfile is cooking from your own pantry: more prep, more decisions, but you choose every flag and patch the recipe wherever you like. The same cook can do either; the choice is how much control you need today.

NIM = prebuilt, optimized, fast, less tuning. Custom Dockerfile = full control over versions, flags, patches. Convenience and peak performance vs flexibility.

06

On your cluster

TL;DR · Your vLLM containers run on OpenShift with an OpenAI API + Prometheus /metrics, GPU via the Container Toolkit, and ~27 GB FP8 weights gated by a careful readiness probe.

Tie it all together on the real box. This is the exact shape of the deployment you've been reasoning about — and the one-paragraph way to teach it back to a teammate.

Your cluster setup

Your vLLM deployment runs as containers on OpenShift, exposing an OpenAI API plus Prometheus /metrics, with GPU access via the Container Toolkit. With ~27 GB FP8 weights, readiness gating is critical — incorrect probe timing causes rollouts to route traffic to cold pods.

Hit the container's OpenAI-compatible API on the 4×H100 box

curl localhost:8000/v1/completions -d '{
  "model": "Qwen3.6-27B-FP8",
  "prompt": "Kubernetes pods are scheduled by the",
  "max_tokens": 16
}'

Cluster note Containers serve Qwen3.6-27B-FP8 (~27 GB FP8 weights) on the 4×H100 box. The image exposes the OpenAI API on :8000 and Prometheus /metrics for scraping; GPU access comes from the NVIDIA Container Toolkit. Because the weights are ~27 GB, the readiness probe is gated on engine-ready state, not process start.

Teach it back

Say it in one breath: "Containerizing our LLM is like our other services except — the pod needs a GPU (nvidia.com/gpu + Container Toolkit), the readiness probe must wait for ~27 GB of weights to load into VRAM (or we route to a dead pod), and the images are huge so we care about layer caching and baked-vs-mounted weights. NIM gives us a prebuilt optimized image; a custom Dockerfile gives control."

Keywords — tap to unfold the plain meaning

Check yourself

What two pieces does a pod need to actually use a GPU?
Why must the readiness probe wait, and what breaks if it passes at process start?
Name the three image concerns that come from weights being multi-GB.
When do you reach for NIM, and when for a custom Dockerfile?

Same containerization as any service, except the GPU (toolkit), the probe (wait for ~27 GB into VRAM), and the huge image. NIM for speed; Dockerfile for control.