Model Formats & Compilation — Inference Engineering

01

Store vs compile: two ways to ship a model

TL;DR · You either serialize a model — write its numbers to disk so any framework can read them — or compile it into a hardware-tuned engine that's blazing fast but welded to one GPU.

Today's win: explain the difference between serializing a model (SafeTensors, ONNX) and compiling one (TensorRT-LLM) — portability vs hardware-tuned performance, and when each is the right call.

Two jobs that look the same

"Save the model" hides two very different acts. Serialization stores the weights (and maybe the graph) portably — any runtime can load them. Compilation rewrites the model into a binary tuned for one exact GPU. Same model, opposite philosophies.

The spectrum, left to right

Formats line up from most portable to most optimized: .pt pickle → SafeTensors → ONNX → TensorRT engine. As you move right you gain speed and lose portability.

Why it matters for serving

The choice decides whether you can swap models on a whim or must rebuild for every change. It's the difference between portability and peak speed — and your cluster has already picked a side.

Left = portable (any framework reads it), right = optimized (one GPU runs it fast). Moving right trades flexibility for speed.

Keywords — tap to unfold the plain meaning

Analogy ❝Serializing a model is writing down a recipe: numbers on paper that any line cook in any kitchen can read and follow. Compiling a model is building a custom assembly line for that one dish — blazing fast, but it only runs in this kitchen on this equipment, and you must tear it down and rebuild it the moment the dish or the equipment changes.

Serialize = write the numbers portably; compile = build a fast, hardware-welded engine. The spectrum runs .pt pickle → SafeTensors → ONNX → TensorRT.

02

SafeTensors: store the weights safely

TL;DR · SafeTensors is a header plus a raw blob — memory-mappable and safe — that stores weights without the supply-chain risk of .pt pickle files.

This is the simplest, most portable end of the spectrum: just the numbers, written down honestly. No hidden code, no graph — exactly what your cluster loads today.

A header plus a raw blob

SafeTensors is a modern serialization format: a small header that names the tensors, followed by one raw blob of weight bytes. That layout makes it memory-mappable — the loader can point straight at the bytes on disk instead of copying them.

Why "safe"?

The old .pt pickle format can execute arbitrary code when loaded — a supply-chain risk. SafeTensors stores only numbers, so opening a file can never run code. Safe to download, safe to load.

What it does not store

SafeTensors holds the weights and nothing else — no computation graph, no math operations. You still need the original framework code that knows how to wire those weights into a model. Portability of numbers, not of the whole network.

Keywords — tap to unfold the plain meaning

Cluster note Your Qwen3.6-27B-FP8 weights on the 4×H100 box ship as SafeTensors in FP8. Because the file is just a header + blob, vLLM mmaps it into GPU memory fast and never risks running embedded code — the flexible, safe default.

SafeTensors = header + raw blob, memory-mappable, safe. It stores weights only — no graph — and avoids the code-execution risk of .pt pickle.

03

ONNX: ship the computation graph too

TL;DR · ONNX stores the weights and the computation graph in a standard operator set, so a model runs across runtimes without its original framework.

SafeTensors gives you numbers but still needs the framework. ONNX goes one step right on the spectrum: it packs the math itself, so the model travels alone.

The graph, written down

A computation graph is the model written as a graph of math operations on tensors — not just its weights. ONNX stores both: the numbers and that graph of operations.

A standard operator set

ONNX expresses the graph in a shared standard operator set — a fixed vocabulary of ops every compliant runtime understands. That's what makes models portable across runtimes without the framework they were trained in.

Still a stored format

ONNX is more complete than SafeTensors but still firmly on the store side: it describes the model, it doesn't yet fuse layers or tune kernels. That final, hardware-specific step is the next station.

SafeTensors carries numbers only; ONNX adds the computation graph in a standard operator set, so it no longer needs the original framework.

Keywords — tap to unfold the plain meaning

ONNX = weights + computation graph in a standard operator set, portable across runtimes without the original framework. Still a store format, not a compile.

04

TensorRT: compile a tuned engine

TL;DR · TensorRT-LLM is a compiler: it fuses layers, picks the fastest kernels, selects precision, and auto-tunes for the exact GPU — producing a binary engine.

Here we cross from storing to building. TensorRT doesn't just describe the model; it rewrites it into machine-tuned code for one specific GPU.

Four things the compiler does

TensorRT / TensorRT-LLM goes beyond storing. It will fuse layers (merge ops to cut overhead), pick the fastest kernels for your hardware, select precision (FP16/FP8), and auto-tune for the exact GPU in the box.

The output is a binary engine

The result is a binary engine — compiled, not described. It's the most-optimized point on the spectrum, the far-right of .pt pickle → SafeTensors → ONNX → TensorRT engine.

The catch: it's welded down

That engine is tuned to one model and one GPU. Change the model, the precision, or the hardware and you must recompile. Peak speed comes welded to a single configuration.

Keywords — tap to unfold the plain meaning

Analogy ❝A stored format is the written recipe — read it in any kitchen. TensorRT is the custom assembly line built around that one recipe: every station fused and placed for the exact equipment in this kitchen, so the dish comes out blazing fast. But it only runs here, on this gear — change the dish or swap a machine and the whole line has to be torn down and rebuilt from scratch.

TensorRT-LLM compiles: fuse layers, pick fastest kernels, select FP16/FP8, auto-tune for the exact GPU → a binary engine. Fast, but welded to one model + one GPU.

05

The tradeoff: flexibility vs peak speed

TL;DR · vLLM with SafeTensors is the flexible path — swap, retune, move, no compile. TensorRT-LLM squeezes more throughput and latency but demands a per-model, per-GPU compile.

This is the whole decision in one sentence: how much flexibility will you trade for how much speed? The right answer depends entirely on your workload.

The flexible path

vLLM on SafeTensors lets you swap models, change flags, and move models between GPUs with no compile step. Iteration is instant; nothing is welded down.

The optimized path

TensorRT-LLM squeezes out more throughput and lower latency — but it demands a per-model, per-GPU compile. Every model change or hardware change means rebuilding the engine.

When compiling is worth it

The compile is worth it only for a frozen, very-high-QPS workload: one model, one GPU type, locked down, served at huge volume. If anything still moves, the flexible path wins.

Keywords — tap to unfold the plain meaning

Flexibility vs peak speed: vLLM + SafeTensors swaps freely with no compile; TensorRT-LLM is faster but per-model, per-GPU — worth it only for a frozen, very-high-QPS workload.

06

On your cluster

TL;DR · Your 4×H100 box runs vLLM with SafeTensors (FP8) — the flexible path — because you swap Qwen versions, retune flags, and move models with no recompile.

This isn't abstract. Your cluster has already made the call, and the reasoning is exactly the tradeoff from Station 05 applied to a real box.

What's running, and why

You run vLLM with SafeTensors (FP8) — the flexible path. You can swap Qwen versions, retune --max-num-seqs, or move models between GPUs with no recompile. The workload still moves, so flexibility wins.

Cluster note On the 4×H100 box: SafeTensors (FP8) + vLLM means a model swap is a config change, not a build. Compiling with TensorRT-LLM would buy throughput/latency, but it welds you to one model + one GPU and must be rebuilt on any change — not worth it while you're still iterating. Day08 notebook reference: pytorch-model-formats.ipynb.

Keywords — tap to unfold the plain meaning

Check yourself

What's the difference between serializing a model (SafeTensors, ONNX) and compiling one (TensorRT-LLM)?
Why do we use vLLM + SafeTensors instead of compiling? (Hint: portable and flexible — swap, retune, move across GPUs with no recompile.)
What would compiling buy you, and what's the cost? (Hint: fuse and auto-tune for peak speed, but welded to one model + one GPU, rebuilt on any change.)
When is a per-model, per-GPU compile actually worth it?

We use vLLM + SafeTensors because serialized formats are portable and flexible — swap, retune, move across GPUs with no recompile. Compiling would fuse and auto-tune for peak speed, but it's welded to one model + one GPU.