RunPod sells the same H100 two ways, which makes it the cleanest natural experiment in this book. A dedicated Pod is $2.89/hour. The serverless tier is $0.00116/second — which is $4.176/hour if you ran it continuously.
So serverless costs 1.44× more per unit of GPU time. The question this chapter answers is when that premium is worth paying, and the answer is not the one the duty-cycle framing suggests.
The premium, priced across the fleet
Same vendor, same silicon tier, two products (rates captured 2026-08-08):
| GPU | Serverless $/sec | Serverless $/hr equiv | Dedicated Pod $/hr | Premium |
|---|---|---|---|---|
| H100 80 GB | $0.00116 | $4.176 | $2.89 (PCIe) | 1.44× |
| A100 80 GB | $0.00076 | $2.736 | $1.39 (PCIe) | 1.97× |
| L40S 48 GB | $0.00053 | $1.908 | $0.99 | 1.93× |
| A6000 / A40 48 GB | $0.00034 | $1.224 | $0.44 (A40) | 2.78× |
Note the spread at the ends: the cheapest GPU carries the steepest premium. An A40 costs 2.78× more served this way; an H100 only 1.44×. Serverless overhead behaves more like a flat cost than a proportional one, so it hurts most where the underlying GPU is cheap. The trend is not strictly monotonic in between — the L40S at $0.99 carries a slightly lower premium (1.93×) than the pricier A100 at $1.39 (1.97×) — so treat this as a shape, not a law, and price your own tier.
The naive crossover, and why it misleads
Dedicated bills every hour whether you use it or not. Serverless bills only while a worker runs. So if u is the fraction of wall-clock time you are actually computing:
Break-even: u* = dedicated rate ÷ serverless hourly rate = 1 ÷ premium
| GPU | Break-even duty cycle |
|---|---|
| H100 | 69.2% |
| L40S | 51.9% |
| A100 | 50.8% |
| A40 | 35.9% |
Read naively that is reassuring: keep an H100 busy less than 69% of the time and serverless wins. That number is wrong for most real workloads, because it assumes every billed second is a computing second. It isn't.
What you actually get billed for
RunPod bills "from when a worker starts until it fully stops, rounded up to the nearest second" — and that window contains three things, only one of which is your work: container start and model load, execution, then the idle timeout, defaulting to 5 seconds, during which "you're billed during idle time, but the worker remains warm."
So for a job with execution time E, cold start C, and idle timeout T, billed seconds per request are C + E + T. The overhead multiplier is:
M = (C + E + T) ÷ E
With a 10-second cold start and RunPod's 5-second default idle timeout:
| Execution time E | Billed per request | Overhead M | Real H100 break-even (69.2% ÷ M) |
|---|---|---|---|
| 1 s | 16 s | 16.0× | 4.3% |
| 2 s | 17 s | 8.5× | 8.1% |
| 10 s | 25 s | 2.5× | 27.7% |
| 60 s | 75 s | 1.25× | 55.4% |
| 300 s | 315 s | 1.05× | 65.9% |
This is the chapter. The crossover is not governed by how much traffic you have — it is governed by how long each job runs. A two-second inference call needs your GPU busy under 8% of the time before serverless beats a dedicated box. A five-minute batch job is at 66%, essentially the naive number.
And the idle timeout alone does damage even when you are perfectly warm. With no cold start at all, M = (E + T) ÷ E: a 1-second job still bills 6 seconds, because the 5-second idle window is charged after every request. Short jobs are where per-second billing quietly stops being per-second.
The fix, where your workload allows it, is not a pricing decision — it is batching. Merging ten 2-second calls into one 20-second call takes M from 8.5× to 1.75×, which moves the H100 break-even from 8.1% to 39.5%.
Modal's headline price is the preemptible price
Modal publishes per-second rates directly — H100 at $0.001097/sec, which is $3.95/GPU-hour, and Modal's own comparison page uses exactly that figure. That looks competitive against RunPod's $4.176 serverless and not far off a dedicated Pod.
Then the plan-comparison table adds two multipliers that never appear in the headline:
- Region selection: "1.5 - 1.75x base prices"
- Non-preemptible execution: "3x base prices"
So an H100 you are guaranteed not to lose mid-job is $11.85/hour — 4.1× RunPod's $2.89 dedicated Pod. The advertised rate buys preemptible capacity in an unspecified region.
That is not a criticism of the pricing; preemptible-by-default is a legitimate design, and spot and interruption explains why it is usually the right trade for restartable work. It is a criticism of the comparison everyone makes. If you are benchmarking serverless against dedicated, you have to compare guarantees, not just numbers — and the guarantee is a 3× line item that lives two tables down the page.
The ceiling nobody checks until it's Friday
Dedicated capacity scales until you run out of money. Serverless scales until it hits a configured cap, and the defaults are low:
| Platform | Concurrency limit |
|---|---|
| RunPod | Max workers default: 3 |
| Modal — Starter ($0) | 100 containers, 10 GPU concurrency |
| Modal — Team ($250/mo) | 5000 containers, 50 GPU concurrency |
RunPod describes max workers as both "a cost safety limit and concurrency cap" and advises setting it "~20% higher than expected max concurrency." Three concurrent workers is the out-of-the-box ceiling — fine for development, an outage waiting for a launch.
Two more defaults worth knowing before they surprise you. Execution timeout is 600 seconds, and when exceeded "the job fails and the worker stops" — a ten-minute wall across your longest job, adjustable from 5 seconds to 7 days but not infinite. And there is a dormancy rule: "After 3 days with no requests, the endpoint's max workers is reduced to 2," and "once an endpoint has been scaled down this way, it stays at its reduced max workers until you raise the value yourself." A quiet endpoint does not come back at full width on its own.
RunPod also applies a default spend limit of $80/hour across all resources, which is a genuinely useful guardrail — and the thing that will stop a runaway fan-out before it stops your card.
The diagnostic
- What is your median job execution time? Under ~10 seconds, per-request overhead dominates and the naive duty-cycle maths is off by multiples. This is the first question, not the fifth.
- Compute M = (C + E + T) ÷ E for your actual numbers. Then divide the naive break-even by it. That is your real crossover.
- Did you choose the idle timeout? Five seconds charged after every request is a large tax on short jobs and a bargain on long ones. It is one field.
- Can you batch? Merging short calls attacks the overhead multiplier directly and is usually the largest single lever available here.
- Are you comparing like guarantees? A preemptible unpinned serverless rate against a dedicated instance is not a comparison. On Modal, matching guarantees costs 3×.
- What is your concurrency cap, and what happens at the cap? RunPod defaults to 3 workers; Modal's Starter tier to 10 GPUs. Find out before your traffic does.
- Is anything relying on an endpoint that has been quiet for three days? It has been throttled to 2 workers and will stay there.
What this chapter is not saying
It is not saying serverless is expensive. For genuinely spiky work with long-ish jobs — batch inference, video generation, scheduled evals — it is the correct answer by a wide margin, and the 1.44× premium on an H100 is trivial against a box idling 90% of the day.
It is saying that "pay only for what you use" is defined by the vendor's billing window, not by your compute, and that on short jobs those two differ by an order of magnitude. The naive break-even says 69%. Your real one might be 4%. The difference is entirely in how long your jobs run, and that is a number you already have.