⚙ Inference Engineering 0 / 36
Part 6 of 6 · Inference Engineering

Zero-Downtime Deployment & Cost

Shipping safely + costing tokens, one guess at a time — the rollout plays, the LLM-specific drain catch, and the cost-per-million-tokens formula that caps the whole course.

Dims everything but the section you're reading.
Color key — each role keeps its own hue Green = where you are / progress Blue = keywords Violet = math Coral = analogy
01 / 05 Ship without dropping a request
  1. 01 Ship without dropping a request
  2. 02 Three rollout plays
  3. 03 The LLM catch: graceful drain
  4. 04 Cost per million tokens
  5. 05 The capstone: worked & checked
01

Ship without dropping a request

TL;DR · A new model or new version has to replace the running one while users keep streaming tokens — no truncated answers, no error spikes, instant escape hatch if it's bad.

This is the capstone of the whole course. You've made the engine fast and routed, scaled, and packaged it — now you have to swap a live model out from under real traffic without anyone noticing, and finally put a price on every token.

The goal: zero downtime

A zero-downtime deployment ships a new version while the old one keeps serving, so no request is dropped and no user sees an error during the switch. The hard part is the in-between moment when both versions, or neither, could be holding a request.

Two outcomes you must control

A safe rollout watches the metrics that matter — P99 latency, error rate, and answer quality — and keeps a fast path back. If the new version regresses, you want an auto-rollback, not a 2 a.m. page.

Keywords — tap to unfold the plain meaning

Analogy ❝A zero-downtime rollout is changing the kitchen's head chef mid-dinner-service. You can't lock the doors and reopen tomorrow — orders are already on the rail and diners are mid-meal. The new chef has to slide in while plates keep going out, and if the first few tickets come back wrong, the old chef steps right back to the pass. Nobody at the tables should ever know it happened.

Zero-downtime means swapping the live model with no dropped requests and no error spike — watch P99/errors/quality, and keep an instant rollback ready.

02

Three rollout plays

TL;DR · Blue-green flips all traffic at once with instant rollback; rolling swaps replicas gradually; canary ramps 1% → 5% → 25% → 100% while watching metrics and auto-rolls-back on regression.

There are only three patterns to know, and they trade safety against how many replicas you have to run at once. Pick by how confident you are and how much spare capacity you can spend.

Blue-green

Blue-green: stand up a full new version (green) alongside the running one (blue), switch traffic over in one flip, and keep the old fleet idle as an instant rollback. Safest reversal, but you pay for two full fleets during the cutover.

Rolling

Rolling: replace replicas gradually, a few at a time, so the fleet is always a mix of old and new. Cheap on capacity, but rollback means rolling back the same slow way.

Canary

Canary: send a trickle of traffic to the new version — 1% → 5% → 25% → 100% — watching metrics at each step, with auto-rollback if P99 latency or error rate crosses a threshold. You catch a bad version while it's only hurting 1% of users.

Three rollout strategies: blue-green flips all traffic at once, rolling replaces replicas a few at a time, and canary ramps traffic 1 to 5 to 25 to 100 percent. blue-green rolling canary old new flip 100% at once keep old = rollback replace a few at a time always a mix 1% 5% 25% 100% ramp + watch metrics auto-rollback on regress safer + more spare capacity → left to right less spend, more risk on the flip
Blue-green flips everything at once and keeps the old fleet as a rollback; rolling swaps replicas a few at a time; canary ramps 1% → 5% → 25% → 100% while watching metrics.

Keywords — tap to unfold the plain meaning

Blue-green = one flip, instant rollback, double the fleet. Rolling = gradual, cheap. Canary = ramp 1/5/25/100% while watching, auto-rollback on regression.

03

The LLM catch: graceful drain

TL;DR · LLM generations run for many seconds, so killing a replica mid-generation truncates the user. Stop admitting new requests, let in-flight ones finish, then exit — via a preStop hook, SIGTERM handling, and a long drain timeout.

This is the one rollout detail that's special to LLMs. A web request finishes in milliseconds; an LLM answer can stream for many seconds. Pull the plug at the wrong moment and you cut someone's answer off mid-sentence.

Why it's different here

Generations can run for many seconds. If a rollout kills a replica mid-generation, those users get a truncated answer. A normal rolling update assumes requests end fast — that assumption breaks for token streaming.

The fix: drain, don't kill

Graceful drain: stop admitting new requests, let in-flight ones finish, then exit. The replica is told "you're going away" but gets time to finish what it's already streaming before it actually stops.

How Kubernetes does it

A preStop hook plus SIGTERM handling triggers the drain, and a drain timeout generous enough for the longest expected output lets it complete. Default terminationGracePeriodSeconds (30s) may be too short — size it to your longest generation.

Graceful drain timeline: a stop signal arrives, the replica stops admitting new requests but lets in-flight generations finish, then exits within the grace period. graceful drain timeline SIGTERM / preStop exit serving normally no new requests admitted — in-flight generations finish long output finishes inside the budget drain timeout = longest expected output (set terminationGracePeriodSeconds > 30s)
On the stop signal, the replica stops admitting new requests but keeps streaming the ones already in flight, exiting only once they finish or the drain timeout expires.

Keywords — tap to unfold the plain meaning

SRE note Wire a preStop hook + long terminationGracePeriodSeconds (the 30s default may be too short for long generations). Size maxSurge/maxUnavailable around minutes-long weight-load readiness, and hook canary analysis to your TTFT / P99 SLOs. Omitted from the simple cost model but real: CPU/RAM, storage and model downloads, networking, idle and failed capacity, control-plane services, licenses, and temporary rollout surge.
Analogy ❝Telling the old chef to leave isn't pulling them off the line mid-plate. You stop seating them new tickets, let them finish the dishes already on their station, and only then do they hang up the apron. Yank them away mid-sear and the diner gets a half-cooked plate — that's a truncated generation. The drain timeout is just how long you'll wait for their longest order to finish.

LLM answers stream for seconds, so never kill mid-generation. Stop new requests, let in-flight ones finish, then exit — preStop + SIGTERM + a drain timeout sized to your longest output.

04

Cost per million tokens

TL;DR · Cost per million delivered tokens = fleet $/hr ÷ (delivered goodput × 3600) × 1,000,000, where delivered goodput = rated capacity × average utilization. The trap: never count utilization twice.

Everything in this course was secretly about this one number. Make tokens cheaper and you do it by serving more tokens per second, or by keeping the fleet fuller. Here's the formula that turns GPU-hours into a price tag.

What "delivered goodput" means

Delivered goodput is the tokens per second you actually hand to users, not the peak the box can hit. It equals rated capacity × average utilization — the rated throughput discounted by how full the fleet really runs.

The double-counting trap

Do not count utilization twice. If tokens/sec came from observed production traffic, it's already a delivered rate — use it directly. If it's a saturation benchmark or rated per-GPU capacity, multiply by expected utilization to estimate delivery.

Math, decoded

$/1M = fleet $/hrGdel × 3600 × 1,000,000
Gdel = Crated × U
  • fleet $/hrtotal dollars per hour to run the whole fleet (e.g. $3 per GPU-hour × number of GPUs)
  • Gdeldelivered goodput — tokens per second you actually deliver to users
  • Cratedrated capacity — the aggregate tokens/sec the fleet can sustain at saturation
  • Uaverage utilization — fraction of capacity actually used (e.g. 0.70 = 70%)
  • × 3600seconds per hour: turns a per-second rate into tokens delivered per hour
  • × 1e6scale the per-token cost up to a price per one million tokens

Take what the fleet costs per hour, divide by how many tokens it actually delivers per hour, then scale to a million tokens. Delivered goodput is rated capacity discounted by utilization — and you apply that discount exactly once.

Keywords — tap to unfold the plain meaning

Analogy ❝Cost per dish = the kitchen's hourly cost ÷ dishes served per hour. A half-empty kitchen still pays full rent, so each plate it does send out costs more. Everything you've learned in this course either serves more dishes per hour — quantization, batching, speculative decoding, better hardware — or keeps the kitchen fuller — routing, autoscaling, packing replicas. Token cost is just the dish price.

$/1M = fleet $/hr ÷ (delivered goodput × 3600) × 1e6, and delivered goodput = rated capacity × utilization. Apply the utilization discount once — never twice.

05

The capstone: worked & checked

TL;DR · $3/GPU-hr, 2,000 tok/s rated, 70% util → 1,400 tok/s delivered → ≈ $0.60 / 1M tokens. If 2,000 was already an observed delivered rate, it's ≈ $0.42 — don't multiply by 0.7 again.

Two levers set the whole price, and one mistake can halve or inflate your estimate. Here's the capstone number, worked both ways so you never count utilization twice.

The two levers

Cost-per-token moves on exactly two dials. tokens/sec rises with quantization, batching, speculative decoding, and hardware selection. Utilization rises with routing, autoscaling, and replica load optimization. Every earlier lesson pushes one of these.

Worked example — rated capacity

Assume $3/GPU-hr, 2,000 aggregate tok/s rated, 70% utilization. Delivered goodput = 2,000 × 0.70 = 1,400 tok/s. Cost ≈ $0.60 / 1M delivered tokens.

The trap, made concrete

If that 2,000 tok/s was an observed delivered rate, it already includes utilization: $3 ÷ (2,000 × 3,600) × 1e6 ≈ $0.42. Do not multiply by 0.7 again — that double-counts the discount.

Capstone cost model — $3/GPU-hr · 2,000 tok/s rated · 70% util

rated cap2000
delivered1400

Rated 2,000 tok/s × 0.70 utilization = 1,400 tok/s delivered → ≈ $0.60 / 1M tokens. If 2,000 was already observed-delivered, it's ≈ $0.42 — apply the 0.70 discount once.

The capstone formula, both readings

# rated capacity case (discount once):
delivered = 2000 * 0.70           # = 1400 tok/s
$/1M = 3 / (1400 * 3600) * 1e6    # ≈ $0.60

# observed delivered rate case (already discounted):
$/1M = 3 / (2000 * 3600) * 1e6    # ≈ $0.42  (do NOT × 0.7 again)

Keywords — tap to unfold the plain meaning

Course arc Model & request foundations (L1–11) → runtime, measurement, quality, diagnosis (L12–23) → precision & formats (L24–26) → multi-GPU scaling (L27–28) → hardware (L29–31) → routed, scaled, shipped, and costed production serving (L32–36). This lesson is the capstone: it ships safely and prices the token.

Check yourself

  1. Name the three rollout plays, and which one ramps 1% → 5% → 25% → 100% with auto-rollback.
  2. Why is killing a replica mid-generation worse for an LLM than for a normal web service — and what three things implement the fix?
  3. Write the cost-per-million-tokens formula. Where does utilization enter, and what's the one mistake that doubles the discount?
  4. Finish it for a colleague: "We ship a new model with no downtime by…"

Canary or blue-green: ramp traffic while watching latency, errors, and quality; rollback on regression; drain in-flight generations within a measured grace budget. Then price the token — discounting utilization exactly once.

Reached the end — nice. This lesson now counts toward your progress.