What an H100-hour actually costs

Once you stop buying tokens and start buying compute, the unit of account changes. You are no longer paying per million tokens; you are renting accelerators by the hour and your cost per token becomes a function of how well you keep them busy.

So the first question is what an hour actually costs. Here is the same silicon — NVIDIA H100 80GB SXM — priced three ways, from published list rates on 2026-08-04:

That is a 2.3× spread on identical hardware, and it is the single largest lever at this layer. Nothing you do to your serving stack will beat starting from the right side of that table.

Incomplete: GCP and Azure. This table prices AWS against two neoclouds. The equivalent SKUs — GCP a3-highgpu-8g and Azure ND96isr_H100_v5, both 8×H100 80GB — belong here and are not yet verified against a live source. They will be added rather than estimated. Until then, treat the hyperscaler column as one data point, not as "the hyperscaler price."

Three things this table says that are not obvious

AWS spot is cheaper than neocloud on-demand. $2.62 against $2.99. The reflex "leave the hyperscaler to save money" is not automatically right — if your workload tolerates interruption, the hyperscaler you are already on may be the cheapest option available, with no migration and no new vendor.

A three-year AWS commitment buys roughly what a neocloud gives you for nothing. Reserved lands at $2.97; RunPod's no-commitment rate is $2.99. You would be signing away three years of flexibility to reach a price you could have had this afternoon on a credit card. That comparison is the one most reserved-instance business cases quietly omit.

Density is priced, and backwards from what you expect. Lambda charges $3.99/GPU/hr for an 8×H100 instance and $4.29 for a single one — you pay more per GPU for fewer of them. There is no volume discount here because the product is not really the GPU; it is the NVLink domain around it. An 8-GPU node is one coherent machine. Eight 1-GPU instances are eight machines that have to talk over the network, which for anything model-parallel is a different and worse product.

The break-even that decides commitments

The reserved-versus-on-demand question has an arithmetic answer, and most teams never compute it.

AWS on-demand is $6.88/GPU/hr; 3-year reserved is $2.97. Reserved wins as soon as:

$2.97 ÷ $6.88 = 43%

That is, if you will keep that GPU busy more than 43% of the hours over three years, reserving is cheaper — and below that, you are pre-paying for idle. The number to test against is not "will we still be using AI in three years" but "will this specific class of accelerator, at this quantity, be busy 43% of the time in year three." Those are very different questions, and the second one is much harder to answer yes to in a market where the hardware generation turns over in roughly two.

Run the same division for your own quotes. It is one line of arithmetic and it converts a strategic-sounding argument into a utilisation forecast you can actually check.

The cheap number is not the whole number

Everything above is list rate for the accelerator. The reason a neocloud can be half the price of a hyperscaler is mostly that it is selling less, and some of what it omits will matter to you:

• Reliability and replacement. A failed node on a hyperscaler is an API call. On a smaller provider it can be a support ticket and a wait. For training runs that checkpoint, this is an annoyance; for production inference it is an availability problem you have to engineer around yourself.
• Egress. Lambda publishes no egress fees. Hyperscalers charge for data leaving, and for inference workloads shipping large responses or moving datasets between providers, that line is real and is not in the GPU rate.
• Capacity guarantees. "On-demand" at a neocloud can mean first-come, first-served. Community-tier capacity in particular is someone else's spare machine, priced accordingly.
• Compliance posture. If you need specific certifications, residency, or a signed BAA, the eligible provider list shortens fast and the price comparison changes with it.
• The rest of the bill. Storage, networking, and the engineering time to run your own serving stack are not in the hourly rate. A team moving off a token API to save on compute frequently discovers it has traded a variable bill for a fixed one plus an on-call rotation.

None of that means the expensive option is correct. It means the honest comparison is total cost of the workload, not the headline hourly rate — and that a 2.3× gap is wide enough that it usually survives an honest accounting, which is exactly why it is worth doing.

Where the rest of the fleet sits

For context, published rates on the same day, per GPU/hr:

Two things to take from this. First, the A100 spread is 87% — $1.49 against $2.79 for the same part. Provider choice matters more than most architectural decisions you will agonise over.

Second, the ladder from A6000 to B300 spans roughly 14×, and the top of it is not always the right answer. The correct accelerator is the cheapest one whose memory holds your model and whose bandwidth meets your latency target — a question about your workload, not about which chip is newest. That is the silicon layer — Part 3 — and it is where this gets genuinely interesting.

Prices are published list rates captured 2026-08-04 from each provider's public pricing page and, for AWS, from the Vantage EC2 index. They move. Configurations differ in vCPU, RAM, storage, and interconnect — the per-GPU figures above are the accelerator slice of a whole-instance price and are comparable in order of magnitude, not to the cent. Verify against a current quote before committing money.