Chapter 2.33 of 12 in this part

Spot, preemptible, and the economics of interruption

AWS gives you two minutes. Azure gives you thirty seconds. Google's default is zero — and Google says outright that if you want warning on a GPU, don't use Spot. None of those windows is long enough to save a loaded model.

7 min read·revised 2026-08-08

Spare capacity sold at a discount is the largest single price lever on the hosting layer, and the discount is not the interesting part. The interesting part is what interruption actually costs you, which is a different number for a batch job than for a model that took four minutes to load.

The three clouds do not offer the same product

All three sell unused capacity. The terms differ in ways that matter operationally:

AWS Spot GCP Spot VMs Azure Spot VMs
Notice period two-minute interruption notice 0 seconds (default), 120 s in Preview 30-seconds notice
Early-risk signal rebalance recommendation preempted metadata flag Scheduled Events, best effort
On interruption "terminates, stops, or hibernates" stops (default) or deletes Deallocate (default) or Delete
Price mechanism "set by Amazon EC2, and is adjusted gradually" "can change up to once every day" max price you set; evicted if exceeded
Published discount not stated on the page read "up to 91%" "significant cost savings"

Two things stand out.

Only Google publishes a percentage in its documentation — "up to 91% discounts for many machine types, GPUs, TPUs, and Local SSDs." AWS's Spot page describes "steep discounts" without a figure; Azure says "significant cost savings." For an actual number you have to price your own instance, which is what hyperscaler versus neocloud does.

The notice periods differ by a factor of four, and Google's default is none at all. AWS's two minutes is the most generous. Azure's 30 seconds arrives via Scheduled Events "on a best effort basis." Google's preemption-notice duration defaults to 0 seconds — there is "no dedicated delay between detecting preemption in metadata and the ACPI G2 Soft Off signal" — with 120 seconds available in Preview, which Google recommends "for any workloads that need a dedicated duration or longer than 30 seconds to handle preemption."

Google says the quiet part about GPUs

The single most useful sentence in any of these documents is Google's advice on GPU workloads:

If you want a warning before your VMs are preempted, or want to configure your VMs to automatically restart after a maintenance event, use standard VMs with a GPU. For standard VMs with GPUs, Compute Engine provides one hour advance notice.

That is the vendor telling you plainly: on Spot GPUs, useful warning is not on offer. It adds that "during maintenance events, Spot VMs with GPUs are preempted by default and cannot be automatically restarted."

One hour versus zero-to-120 seconds is not a difference of degree.

And this is where autoscaling accelerators collects its debt. Getting tens of gigabytes of weights into GPU memory is measured in minutes, not seconds. No notice period any cloud offers is long enough to preserve a loaded model. You can flush a checkpoint. You cannot save residency. Every interruption means paying the cold start again — and that cold start is billed.

What interruption actually costs, computed

Three quantities decide whether the discount survives contact with your workload:

  • T — mean uptime between interruptions
  • R — restart cost: reacquire capacity, reload weights, warm up
  • C — checkpoint interval, so expected lost work per interruption is about C/2

Paid time per unit of useful work is (T + R + C/2) / T. At a spot price of 0.10× on-demand, effective cost per useful hour:

Uptime T Restart R Checkpoint C Effective vs on-demand
4 h 5 min 20 min 0.106× (89% off)
1 h 5 min 20 min 0.125× (87.5% off)
30 min 5 min 20 min 0.150× (85% off)
10 min 5 min 20 min 0.250× (75% off)

The finding is not the one people expect. Spot is remarkably robust to interruption for restartable work. Even at ten-minute uptimes with a five-minute reload — a punishing regime — you are still at a quarter of on-demand. The discount is deep enough to absorb a lot of thrash.

Which reframes the decision: for batch, the interruption rate almost never makes spot a bad trade. Checkpoint frequently and stop worrying. The three published mitigations all help here — AWS's persistent requests automatically resubmit after interruption; Google lets you check "expected uptime" and "historical preemption rate" before you create; Azure's Deallocate policy lets you redeploy later.

Where it genuinely fails

The model above prices delay. It does not price unavailability, and that is the real boundary.

Azure states the constraint in one line: "there's no SLA for these VMs. An Azure Spot Virtual Machine offers no high availability guarantees." An interrupted serving endpoint does not run slower — it drops requests. That is not a cost problem wearing a latency costume; it is an availability problem, and the cost framing hides it.

So the honest split is not "cheap versus expensive." It is:

  • Restartable work — training, batch inference, evals, data prep, dev environments. Spot is close to a free lunch, and the arithmetic above says so even under bad conditions.
  • Serving user traffic — spot is a capacity supplement behind on-demand baseline, never the baseline itself.

Both AWS and Azure name the same workload list: AWS says Spot is "well-suited for data analysis, batch jobs, background processing, and optional tasks"; Azure says "batch processing jobs, dev/test environments, large compute workloads." Neither lists serving. That omission is the recommendation.

Four traps in the fine print

  1. Stopped is not free. Google: a stopped preempted VM incurs no VM hours, but attached persistent disks "still incur storage charges until you delete them." Azure's Deallocate policy is explicit that deallocated VMs "count against your quota and you'll be charged storage costs for the underlying disks."
  2. Redeployment is not guaranteed. Azure: "there's no guarantee that the allocation will succeed." Capacity you released may not come back when you want it.
  3. Premium OS licences don't discount. Google: "Spot VMs don't reduce the cost of premium operating systems."
  4. You cannot debug the interruption. Google: "You can't view why any given preemption event occurred." Rates "vary in different zones, at different times, or with different resources."

One genuinely generous rule worth knowing: Google does not charge for GPUs if their VMs are preempted in the first minute after starting. That directly limits the worst case — thrash where you never get past the load.

The diagnostic

  1. Is this work restartable, or is a user waiting? This single answer decides the chapter. Everything else is tuning.
  2. What is your R — reacquire plus reload? For a large model this is minutes, and it is billed. Measure it before modelling anything.
  3. What is your checkpoint interval? Expected lost work is about half of it. This is the one term you fully control.
  4. Have you checked the published preemption rate? Google exposes expected uptime and historical rates per machine type and zone before you create. Use it.
  5. Are you paying for stopped-instance disks? Deallocate and stop both leave storage billing running.
  6. Is your baseline on-demand? Spot as supplement is sound engineering. Spot as the floor under user traffic is an outage you have scheduled but not dated.
Sources & methodcaptured 2026-08-08

Sources, captured 2026-08-08: the two-minute interruption notice, the rebalance-recommendation signal, the "terminates, stops, or hibernates" behaviour, persistent versus one-time requests, the "set by Amazon EC2, and is adjusted gradually based on the long-term supply of and demand" pricing mechanism, and the suitable-workload list are quoted from AWS's EC2 Spot Instances user-guide page. The "up to 91% discounts for many machine types, GPUs, TPUs, and Local SSDs" figure, the preemption-notice-duration options of 0 seconds (default) and 120 seconds (Preview) with the recommendation quoted, the ACPI G2 Soft Off mechanism, the GPU guidance to use standard VMs for advance warning with "one hour advance notice," the no-charge-for-GPUs-preempted-in-the-first-minute rule, "Spot prices can change up to once every day," the persistent-disk storage charge on stopped VMs, "You can't view why any given preemption event occurred," the expected-uptime and historical-preemption-rate tooling, and the premium-operating-systems exclusion are quoted from Google's Compute Engine Spot VMs documentation. The "30-seconds notice," "there's no SLA for these VMs," "no high availability guarantees," the Deallocate/Delete eviction policies with their quota and storage consequences, the max-price eviction condition, the best-effort Scheduled Events delivery, and the workload list are quoted from Microsoft's Azure Spot Virtual Machines page. The interruption-economics table is entirely mine — computed from the stated model (T + R + C/2) / T applied to an assumed 0.10× spot price, and verified in a separate pass. The 0.10× spot price is an illustrative assumption, not a measured rate, chosen as a round stand-in near Google's published 91% ceiling; your actual discount must come from pricing your own instance. T, R and C are inputs you must measure, not figures taken from any vendor — no cloud publishes a mean-time-between-interruptions, and Google explicitly says preemption rates vary by zone, time and resource. The claim that no notice period is long enough to preserve a loaded model follows from combining these notice periods with the weight-loading times sourced in autoscaling accelerators; that combination is my inference, not a statement any vendor makes. The diagnostic is my framing.

Want this done on your account rather than by you?

The handbook is the method, written out in full so you can run it yourself — that is the point of publishing it. If you would rather someone else did the first pass, the teardown is free and you keep the findings either way.