Multi-Cloud Capacity — Inference Engineering

01

Why inference fleets span clouds

TL;DR · You go multi-cloud for three reasons: GPU supply, lower latency, and surviving any one cloud's outage.

Lesson 34 containerized the engine. Now we place those containers across clouds — and the "why" comes down to three drivers you can name on one hand.

Driver 1 — Supply

GPUs are scarce. You chase capacity wherever it exists: across hyperscalers (the big public clouds), neoclouds (GPU-specialist providers), and resellers. No single provider can guarantee all the H100s you want, so you spread the hunt.

Driver 2 — Latency

Place replicas near your users to cut round-trip time. A request that crosses an ocean pays for every kilometre; a replica in the right region answers from next door.

Driver 3 — Reliability

No single cloud or region should be a single point of failure. If one provider's region goes dark, traffic survives because another region is already serving.

Keywords — tap to unfold the plain meaning

Analogy ❝Think of a restaurant chain that can't get enough cooks in one city. So it opens branches across many cities: it hires from whoever has cooks to spare (supply), it seats each diner at the branch nearest them so the food is hot when it arrives (latency), and if one branch loses power, the others keep serving dinner (reliability). Same menu everywhere — just more kitchens, closer to more diners.

Three drivers, one hand: supply (scarce GPUs), latency (replicas near users), reliability (no single point of failure).

02

One brain, many planes

TL;DR · A single global control plane (policy, routing, capacity) sits over per-cloud workload planes that actually run the GPUs.

The whole multi-cloud shape fits in one line — and that line is worth memorizing, because it's the answer to "explain it to a colleague."

The control plane decides

One global control plane holds the policy, routing, and capacity state across every cloud. It's the brain: it knows where capacity is, where requests should go, and what the rules are. It doesn't run a single GPU itself.

The workload planes do

Each cloud or region has its own workload plane — the place that actually runs the GPUs and serves inference. There are many of them; they're the hands, not the brain.

The one-line shape

Say it out loud: a global control plane (policy, routing, capacity) over per-cloud/region workload planes that run the GPUs, with geo-aware load balancing steering each request to the nearest healthy region in its latency budget.

One control plane on top, many workload planes below — across hyperscalers, neoclouds, and your own on-prem box.

Keywords — tap to unfold the plain meaning

Analogy ❝The restaurant chain has one head office and many branches. The head office never plates a single dish — it sets the menu, decides which branch gets which diners, and tracks who has open tables. The branches do the cooking. One brain coordinating many kitchens: that's the control plane over its workload planes.

One global control plane (policy, routing, capacity) over per-cloud/region workload planes that run the GPUs. Brain decides; hands serve.

03

Geo-aware routing steers by RTT

TL;DR · Geo-aware load balancing sends each request to the nearest healthy region within its latency budget, using round-trip time as the metric.

"Near the user" isn't a vibe — it's a number. The control plane measures round-trip time and routes to the closest region that's both healthy and inside budget.

Nearest healthy region

Geo-aware load balancing sends each request to the nearest healthy region — nearest by round-trip time (RTT), healthy meaning it's actually up and serving. Distance alone isn't enough; a closer-but-down region gets skipped.

Within the latency budget

Each request has a latency budget: a per-region RTT limit beyond which a region isn't allowed to take the traffic. If the closest region blows the budget, routing falls to the next one that fits.

Active-active vs active-passive

For surviving a region loss, you choose a failover stance: active-active (multiple regions serve live at once) or active-passive (a standby region waits to take over). Both keep traffic alive when one region drops.

Math, decoded

route → min_r RTT_r subject to healthy(r) ∧ RTT_r ≤ budget

RTT_rround-trip time from the user to region r — how long a request-and-reply takes
min_rpick the region with the smallest RTT — the nearest one
healthy(r)the region must actually be up and serving, not just close
≤ budgetits RTT must fit inside the request's latency budget, or it's disqualified

In words: send the request to the lowest-RTT region that is both healthy and inside the latency budget. Closer wins, but only if it's up and fast enough.

Keywords — tap to unfold the plain meaning

Analogy ❝The head office seats each diner at the nearest branch — but only if that branch is open and can serve within the time the diner is willing to wait. The branch across the street is useless if its lights are off or its kitchen is backed up past your patience; you get sent to the next-closest one that's actually open and fast enough.

Geo-route by RTT to the nearest healthy region inside the latency budget. Closest, but only if up and fast enough.

04

Procurement trades cost for certainty

TL;DR · Mix three purchase types — reserved baseline + on-demand burst + spot for slack — to balance cost against certainty.

You don't buy GPUs one way. You blend three, each trading price against how sure you are it'll still be there in five minutes.

Reserved — your baseline

Reserved capacity is committed up front: cheapest per hour, but you're locked in. Use it for baseline load — the steady demand you know you'll always have.

On-demand — your burst

On-demand is flexible and the priciest per hour. You pay the premium for bursts — the spikes you can't predict and don't want to commit to year-round.

Spot — your slack

Spot / preemptible is the cheapest of all, but it can be reclaimed any time. Use it for interruptible or buffered work — the slack that can absorb a sudden eviction.

Math, decoded

cost: spot < reserved < on-demand | fleet = reserved + on-demand + spot

spotcheapest per hour — but can be reclaimed at any moment
reservedcheapest committed price — locked in for the baseline you always run
on-demandpriciest — the flexible option you pay extra for during bursts
fleeta typical mix: reserved baseline + on-demand burst + spot for slack

Cheapest to priciest per hour: spot, then reserved, then on-demand. A typical fleet is reserved baseline + on-demand burst + spot for slack — certainty where you need it, savings where you can absorb risk.

Keywords — tap to unfold the plain meaning

Analogy ❝Staffing the kitchen: full-time cooks are your reserved baseline — committed and cheap per shift, always there for the dinner rush you know is coming. On-call cooks you ring in for a busy Saturday are on-demand: flexible, but you pay a premium. And the eager culinary student who'll work for almost nothing but might get pulled to class mid-shift is spot — cheapest of all, but you only give them work that can be dropped.

Reserved baseline + on-demand burst + spot slack. Cost order: spot < reserved < on-demand. Certainty costs money; slack saves it.

05

Cost and resilience are dials; compliance is the fence

TL;DR · You tune cost and resilience freely, but compliance — SOC 2, HIPAA, data residency — is a hard boundary on where workloads can run.

Most of multi-cloud is a tuning problem. But one part isn't negotiable: some workloads simply may not run in some places. That's the fence around the dials.

The dials you turn

Cost and resilience are the dials. Want more savings? Lean on spot and a leaner footprint. Want more resilience? Spread across more regions and run active-active. You're free to slide between them.

The fence you can't cross

Compliance is the fence: SOC 2, HIPAA, and data residency requirements bound where a workload may be placed. A region outside the allowed set is off-limits no matter how cheap or close it is.

Keywords — tap to unfold the plain meaning

Analogy ❝The restaurant can choose how cheaply it staffs and how many backup branches it keeps open — those are dials it turns at will. But health codes say which kitchens may handle certain food at all. You can be as thrifty or as redundant as you like inside the fence; you simply cannot serve outside it.

Cost and resilience are the dials; compliance (SOC 2, HIPAA, data residency) is the fence. Tune inside it, never outside.

06

On your cluster

TL;DR · You're single-site today (on-prem OpenShift, 4×H100). The scale-out path: keep on-prem as the reserved baseline and burst to cloud.

You don't need all of this yet — single-site is fine for now. But the growth path is already clear, and it reuses every term in this lesson.

Where you are

You're single-site: on-prem OpenShift, 4×H100. That's fine for now — one workload plane, no control plane needed yet.

The scale-out path

When you outgrow it: keep on-prem as the reserved baseline and burst to cloud (on-demand/spot) under a global control plane with geo-aware routing. The same pattern fits any service "now sized in GPUs."

4×H100 note Your current single-site cluster is on-prem OpenShift with 4×H100. The clean growth move is on-prem-baseline + cloud-burst: on-prem stays the cheap committed floor (reserved), and overflow spikes spill to the cloud as on-demand or spot under one control plane that geo-routes by RTT.

On-prem 4×H100 holds the baseline; only the spikes above it spill to cloud on-demand/spot — cheap floor, elastic ceiling.

Keywords — tap to unfold the plain meaning

Check yourself

Explain to a colleague: "We'd go multi-cloud for inference to…" (name the three drivers).
Recall the multi-cloud shape in one line — control plane, workload planes, and how requests get routed.
Order the three purchase types by cost, and say which one you'd use for baseline, burst, and slack.

Single-site is fine now; the path is on-prem reserved baseline + burst to cloud under one geo-routing control plane — every term in this lesson, reused.