Chapter 4.66 of 9 in this part

Autoscaling accelerators

The cold start is not a latency problem you tolerate. It is a billed line item you pay for on every scale-up — and AWS's serverless inference does not offer GPUs at all.

10 min read·revised 2026-08-08

Scale-to-zero is the most attractive idea in infrastructure: pay only when someone is asking. For accelerators it is also the hardest, because the thing you must do before serving a request — get tens of gigabytes of weights into GPU memory — takes long enough that you can watch it.

This chapter is about what that actually costs, and it is worth stating the finding up front because it is usually framed wrong. The cold start is not just latency you absorb. It is compute you are billed for.

You pay for the loading

RunPod's serverless pricing documentation lists the phases a worker is charged for, and the first one is the giveaway:

Workers incur charges during three phases:

  1. Start time: Initializing the container and loading models into GPU memory.
  2. Execution time: Processing requests.
  3. Idle timeout duration: The time a worker remains active (running) after completing a request, waiting for additional requests before scaling down (default: 5 seconds).

Only the middle one does work for a user. Billing runs "from when a worker starts until it fully stops, rounded up to the nearest second."

Modal states the same thing from the other direction, about keeping containers alive rather than starting them:

Increasing the scaledown_window reduces the chance that subsequent requests will require a cold start, although you will be billed for any resources used while the container is idle (e.g., GPU reservation or residual memory occupancy).

And then names the trade explicitly: these strategies "introduce a trade-off between cold start latencies and cost."

That is the whole chapter, admitted by both vendors. Scale-to-zero does not eliminate idle cost — it converts idle cost into start-up cost, and whether that is a win depends entirely on your request pattern.

The default idle windows differ by 12×

Two platforms, two very different opinions about how long to hold a GPU after a request finishes:

Platform Default idle-before-scaledown Configurable range
RunPod serverless 5 seconds endpoint setting
Modal 60 seconds 2 seconds to 20 minutes

Modal's default holds the accelerator 12× longer. Neither is wrong — they are different bets about whether the next request arrives soon. But if you moved a workload between them and changed nothing, your idle burn per request would change by that factor, and nothing in your code would explain it.

Modal also exposes min_containers (a floor, so the service never reaches zero) and buffer_containers (spare capacity while active). Both are, in cost terms, a decision to pay for idle GPUs deliberately rather than pay for cold starts repeatedly.

What a cold start actually consists of

Modal's documentation separates the two costs, which is a useful distinction because they have different fixes:

  1. inputs may spend more time waiting in a queue for a container to become ready
  2. there may be extra work that only needs to be done on the first invocation ("initialization")

On the infrastructure half they are quick — "Containers boot in about one second." On the total, they are honest: the time to become warm and ready "can range from seconds to minutes."

The gap between one second and several minutes is your model. Their guidance is unambiguous about the fix and its size:

you might be downloading a large model from a model server during the boot process. You can instead download the model ahead of time … For models in the tens of gigabytes, this can reduce boot times from minutes to seconds.

Two further levers are documented: loading multiple large files concurrently rather than sequentially, and memory snapshots, which capture a warmed container's memory and reuse it on later boots.

The container is not your cold start. The weights are.

The cloud comparison, which has a hard edge

Here the three hyperscalers do not offer the same product at all, and one of them does not offer it in any usable form for LLM work.

AWS — SageMaker Serverless Inference does not do GPUs. Not a caveat, a documented exclusion:

Some of the features currently available for SageMaker AI Real-time Inference are not supported for Serverless Inference, including GPUs, AWS marketplace model packages, private Docker registries, Multi-Model Endpoints, VPC configuration, network isolation, data capture, multiple production variants, Model Monitor, and inference pipelines.

The memory ceiling settles it independently. Serverless endpoints run from 1,024 MB to a maximum of 6,144 MB (6 GB) of RAM, and the docs advise that "the memory size should be at least as large as your model size." A 7B model at 16-bit weights is about 14 GB — more than twice the maximum — before you consider that there is no accelerator to put it on. For AWS, scale-to-zero GPU inference means building it yourself on something else, or using real-time endpoints that do not scale to zero.

GCP — Cloud Run does, and publishes a startup number. GPU on Cloud Run offers NVIDIA L4 (24 GB VRAM) and RTX PRO 6000 Blackwell (96 GB VRAM), and:

Instances of a Cloud Run service that has been configured to use GPU can scale down to zero for cost savings when not in use.

Cloud Run instances with an attached NVIDIA RTX PRO 6000 Blackwell GPU or L4 GPU with drivers pre-installed start in approximately 5 seconds, at which point the processes running in your container can start to use the GPU.

Note what that 5 seconds covers — drivers ready, GPU usable. It does not include loading your weights, which is the part Modal measures in minutes. One GPU per instance; the Blackwell option requires a minimum of 20 CPU and 80 GiB of memory.

Azure — GPU yes, and a different scaling model. Azure ML managed online endpoints "work with powerful CPU and GPU machines in Azure in a scalable, fully managed way," so the accelerator is available. But the shape differs from Cloud Run's: a deployment declares an instance type and an instance count — "the number of instances to use for the deployment. Base the value on the workload you expect" — and autoscaling runs through Azure Monitor, which "flexibly provides between min and max instances, depending on rules." That is instance-count autoscaling over declared VMs, not request-driven scale-to-zero.

Every example in Microsoft's own autoscale documentation floors at two instances. The scale-in rule is described as releasing "a single node, down to a minimum of two," and the scheduled-profile example sets --min-count 2 --count 2 --max-count 2.

That is consistent with a nonzero floor but does not prove one — documented examples show practice, not constraints, and Microsoft never states whether the minimum may be set to zero. So the declaration stands: GPU available, scaling model confirmed, floor still unconfirmed. The practical upshot is unchanged either way — Azure's documented pattern keeps instances running, which is architecturally the opposite of Cloud Run's request-driven scale-to-zero.

Independents — RunPod (flex workers "scale to zero when idle" versus active workers "always running (24/7)") and Modal, both quoted above, are the ones publishing per-second GPU billing and cold-start mechanics in this much detail.

The provisioned-concurrency escape hatch, and its bill

Every platform offers the same out: pay to keep something warm. AWS's version is the clearest about what you are buying and what you are charged for:

Provisioned Concurrency allows you to deploy models on serverless endpoints with predictable performance … SageMaker AI ensures that for the number of Provisioned Concurrency that you allocate, the compute resources are initialized and ready to respond within milliseconds.

You also pay for Provisioned Concurrency usage, based on the memory configured, duration provisioned, and the amount of concurrency enabled.

Read that second sentence carefully. You are billed for provisioned duration whether or not requests arrive. That is not serverless pricing with a latency improvement bolted on; it is reserved capacity wearing a serverless API. Which is fine — reserved capacity is the right answer for steady traffic — but it should be chosen knowingly, because at that point you are comparing against a plain instance, not against scale-to-zero.

The diagnostic

  1. What is your inter-request gap distribution? Not the mean — the distribution. If most gaps are shorter than your idle window you are nearly always warm; if most are longer you are paying start-up on nearly every request. This one number decides everything else.
  2. How long does your cold start actually take, split into container versus weights? Modal boots containers in ~1 second; if yours takes two minutes, ~119 of those seconds are your model and that is where the fix is.
  3. Are your weights baked in or downloaded at boot? Pre-staging is documented to take tens-of-gigabytes loads "from minutes to seconds." It is usually the largest single win available here.
  4. What is your idle timeout, and did you choose it? The defaults differ by 12× across platforms. Inheriting one is a cost decision you did not make.
  5. If you need a GPU and scale-to-zero, have you checked your cloud actually offers both? On AWS SageMaker Serverless, it does not.
  6. Is provisioned concurrency cheaper than an instance? Once you are paying for provisioned duration, compare against reserved or committed-use instance pricing, not against zero.

What this chapter is not saying

It is not saying avoid scale-to-zero. For genuinely spiky, low-duty-cycle work — an internal tool, a batch trigger, a demo — paying start-up a few times a day beats renting an idle H100 around the clock by an enormous margin, and the arithmetic is not close.

It is saying the break-even is a real number that depends on your traffic shape, and almost nobody computes it. The marketing framing is "pay only for what you use." The billing reality is that you also pay for the loading and the waiting, and on accelerators those are not rounding errors.

Sources & methodcaptured 2026-08-08

Sources, captured 2026-08-08: the three billed phases (start time / execution time / idle timeout), the 5-second default idle timeout, the per-second billing "from when a worker starts until it fully stops," and the flex-versus-active worker distinction are quoted from RunPod's serverless pricing documentation, which also lists container disk at approximately $0.10/GB/month and network volumes at $0.07/GB/month under 1 TB and $0.05/GB/month above. The "billed for any resources used while the container is idle," the "trade-off between cold start latencies and cost," the ~1-second container boot, the "seconds to minutes" warm-ready range, the "from minutes to seconds" result of pre-downloading tens-of-gigabytes models, the 60-second default scaledown_window with its 2-second-to-20-minute range, and the min_containers / buffer_containers / memory-snapshot mechanisms are quoted from Modal's cold-start performance guide. The GPU feature exclusion, the 1,024–6,144 MB memory range, the "memory size should be at least as large as your model size" guidance, the tolerate-cold-starts framing, and the Provisioned Concurrency description including its billing basis are quoted from AWS's SageMaker Serverless Inference documentation. The scale-down-to-zero statement, the ~5-second GPU-ready startup, the L4 24 GB and RTX PRO 6000 Blackwell 96 GB VRAM options, the one-GPU-per-instance limit and the Blackwell 20-CPU/80-GiB minimum are quoted from Google's Cloud Run GPU documentation. The statement that Azure ML managed online endpoints "work with powerful CPU and GPU machines," the instance-type/instance-count deployment attributes with the "base the value on the workload you expect" guidance, and the Azure Monitor autoscale description providing "between min and max instances" are from Microsoft's online-endpoints concept documentation. The "down to a minimum of two" scale-in description and the --min-count 2 --count 2 --max-count 2 scheduled-profile example are from Microsoft's Autoscale online endpoints how-to. Azure's minimum-instance floor remains explicitly declared unsourced rather than estimated. Three passes have now been spent on it: pass one found the GPU support, pass two the instance-count scaling model, pass three that every documented example floors at two. None of that establishes whether the minimum may be zero, because example values demonstrate practice rather than constraints — and inferring a limit from a pattern of examples is exactly the move this book's provenance discipline exists to prevent. It stays declared unless Microsoft states the constraint directly. The computed figures are mine: the 12× ratio between Modal's 60-second and RunPod's 5-second idle defaults, and the ~14 GB 16-bit weight footprint of a 7B model being more than twice SageMaker Serverless's 6 GB ceiling, both verified in a separate pass. The weight footprint uses the standard two-bytes-per-parameter approximation for 16-bit weights and excludes KV cache and activations, so it understates real requirements. No per-GPU-hour prices are quoted here, because the platforms publish those on separate pricing pages that render client-side and this chapter's argument does not depend on them — the rates belong in what an H100-hour costs. The diagnostic is my framing.

Want this done on your account rather than by you?

The handbook is the method, written out in full so you can run it yourself — that is the point of publishing it. If you would rather someone else did the first pass, the teardown is free and you keep the findings either way.