Shipping safely + costing tokens, one guess at a time.
You need to ship a new model version without dropping an in-flight stream. For the cost exercise, assume one H100 costs $3/GPU-hour, its SLO-valid rated capacity for the target input/output mix is 2,000 aggregate tokens/s, and average delivered load is 70% of that capacity.
Deployment rolling updates / Argo Rollouts canary, with a preStop hook + long terminationGracePeriodSeconds so pods finish in-flight generations (the default 30s may be too short), and maxSurge/maxUnavailable sized around the minutes-long weight-load readiness. Canary analysis hooks to your TTFT/P99 SLOs (L15).
Canary new Qwen versions with rollback on TTFT, TPOT, errors, and quality checks; drain for the longest supported generation; and budget temporary surge capacity for weight loading. If 2,000 tok/s is the rated capacity, delivered goodput is 2,000 × 0.7 = 1,400 tok/s, so $3 ÷ (1,400 × 3,600) × 1e6 ≈ $0.60/1M delivered tokens. If 2,000 tok/s were already an observed delivered rate, multiplying by 0.7 again would double-count utilization. Your Lab →
Ship model updates with no downtime, and put a dollar figure on every token.
Swapping in a new model version is changing the crew during dinner rush without stopping service. Bring the new line up before retiring the old (blue-green), or swap a few cooks at a time (rolling/canary) while watching complaints (P99). The catch unique to you: a cook mid-dish (an in-flight generation) must be allowed to finish the plate, not be yanked off.
| new line up before old comes down | blue-green / rolling / canary |
| watch complaints, revert if bad | auto-rollback on P99 / errors |
| let the cook finish the plate | graceful drain of in-flight generations |
| cost per dish | $ / 1M tokens |
Three standard plays:1 blue-green (stand up a full new version, switch traffic, keep the old as instant rollback); rolling (replace replicas gradually); and canary (send 1% → 5% → 25% → 100%, watching metrics, with auto-rollback if P99 latency or error rate crosses a threshold).
Generations can run for many seconds. If a rollout kills a replica mid-generation, those users
get truncated. So on shutdown you must drain gracefully: stop admitting new requests, let
in-flight ones finish, then exit, via a preStop hook, SIGTERM handling,
and a drain timeout generous enough for the longest expected output. And remember: each new
replica has a minutes-long cold start, so rollouts are
inherently slow. Plan the window.1
It's Deployment rolling updates / Argo Rollouts canary; you know this. Two GPU
twists: set a preStop hook + long terminationGracePeriodSeconds so a pod
finishes its in-flight generations before SIGKILL (the default 30s grace may be shorter than your tail); and
size maxSurge/maxUnavailable around the minutes-long weight-load readiness
so you don't drop capacity mid-rollout. Canary analysis hooks straight to your TTFT/P99 SLOs from Lesson 15.
The number that ties it all to the business:2
Do not count utilization twice. If tokens/sec came from observed production traffic,
it is already a delivered rate: use it directly. If it is a saturation benchmark or rated per-GPU capacity,
multiply by expected utilization to estimate delivery. Keep the token definition explicit—input, output, or
aggregate at a stated mix—and count only tokens meeting the SLO if the business metric is goodput.
The levers are exactly what this course has been about: tokens/sec (quantization L24, batching L18, speculative decoding L21, the right hardware L30) and utilization (routing L32, autoscaling L33, keeping replicas near the knee L15). Halve the bytes or double the batch and the cost per token drops with it.
Roll out new Qwen versions with a canary + auto-rollback on TTFT/P99,
a preStop drain long enough for your longest generation, and surge sizing that accounts for the
27 GB weight-load readiness. Worked assumption: $3/GPU-hr, 2,000 aggregate tok/s rated
capacity for the target input/output mix, and 70% average utilization. Delivered goodput is
1,400 tok/s, so cost is ≈ $0.60 / 1M delivered tokens. If 2,000 tok/s is already the observed
delivered rate, the answer is $3 ÷ (2,000 × 3,600) × 1e6 ≈ $0.42; do not multiply by 0.7 again.
What this omits: CPU/RAM, storage and model downloads, networking, idle and failed capacity, control-plane services, licenses, and temporary rollout surge. Include them for a fleet TCO rather than a GPU-only serving estimate.
That's the whole arc: model and request foundations (L1–11) → runtime, measurement, quality, and diagnosis (L12–23) → precision and formats (L24–26) → multi-GPU scaling (L27–28) → hardware (L29–31) → routed, scaled, shipped, and costed production serving (L32–36). You started in the ops layer; now you can see all the way down to the silicon and back up to the bill. · Your Lab →
preStop, signals, and grace-period behavior; day26
lab companion.