Memory bandwidth is the product

Below the hourly rate is a physical machine, and two of its numbers set the ceiling on everything above: how fast it can read memory, and how much memory it has. Compute is rarely the binding constraint on inference. Almost everyone shops for it anyway.

NVIDIA published the cleanest possible proof of this, and it is sitting in their own spec sheets.

The controlled experiment

Here are the H100 and H200, from NVIDIA's product pages:

Every compute number is identical. Same power envelope. Same interconnect. The H200 is an H100 with more memory, faster.

NVIDIA's own benchmark for the pair: 1.9× the throughput on Llama2 70B inference.

Nineteen-tenths the work out of zero extra FLOPS. If compute were the constraint, that number would be 1.0.

Why: decode reads the whole model, every token

Generating one token requires streaming every active parameter from memory into the compute units. Not once per request — once per token. The GPU spends most of decode waiting on memory, with its arithmetic units substantially idle. That is the prefill/decode asymmetry seen from underneath.

This gives you a hard speed limit you can compute yourself. A 70B model at FP16 is roughly 140 GB of weights. Divide bandwidth by weight bytes:

• H100: 3.35 TB/s ÷ 140 GB ≈ 24 tokens/second
• H200: 4.8 TB/s ÷ 140 GB ≈ 34 tokens/second

That is the ceiling for a single sequence, and no amount of compute moves it. The ratio, 1.43×, is exactly the bandwidth ratio — because bandwidth is the mechanism.

Two things follow immediately:

Quantization is a bandwidth optimisation, not a compute one. Drop that 70B model to FP8 and the weights halve to ~70 GB. The ceiling doubles: ~48 tok/s on H100, ~69 tok/s on H200. You did not add compute. You made the thing it was waiting for smaller.

Batching is the only way to climb. Weights read for one sequence can serve every other sequence being decoded in the same step. Batch 32 sequences and you read the model once for 32 tokens instead of once each — which is why continuous batching is the single largest serving lever, and it is a memory-bandwidth argument.

The number that explains the whole layer

Divide compute by bandwidth and you get arithmetic intensity — the FLOPs you must perform per byte read to keep the compute units fed:

• H100: 1,979 TFLOPS ÷ 3.35 TB/s = 591 FLOPs per byte
• H200: 1,979 TFLOPS ÷ 4.8 TB/s = 412 FLOPs per byte

Now compare that to what decode actually does. At batch size 1, decode performs roughly one multiply-add per weight — about 2 FLOPs per byte.

Two, against a machine that needs 591 to break even.

At batch size 1 you are using well under 1% of the compute you are renting. You are paying for a Formula 1 engine to idle in traffic.

Arithmetic intensity rises roughly linearly with batch size, so the crossover lands around batch ~300 on an H100 before compute becomes the binding constraint. Below that — which is where nearly all real serving lives — you are on the memory-bound side of the roofline, and every decision should follow from that.

It also explains why the H200's lower arithmetic intensity is a feature. 412 is an easier target than 591. The H200 is better balanced for inference precisely because its compute is less over-provisioned relative to its memory.

What this means for choosing a chip

The right accelerator is the cheapest one whose memory holds your model and whose bandwidth meets your latency target. Not the newest, and not the one with the biggest FLOPS number on the slide.

Work it in this order:

1. Will the weights fit? Parameters × bytes-per-parameter, plus KV cache, plus activation headroom. If it does not fit, nothing else matters — you are into multi-GPU and paying an interconnect tax.
2. Does bandwidth ÷ weight-bytes clear your tokens-per-second target? One division. If it does not clear at batch 1, no serving trick saves you at low load.
3. Only then compare price.

Teams routinely invert this and buy on FLOPS, which is the one number that is usually not binding.

The trade this makes concrete right now

Put the specs against the prices. On GCP, from the accelerator-optimized pricing page on 2026-08-05:

The H200 instance is cheaper than the H100 instance — 4% less per GPU — while delivering 1.9× the inference throughput on NVIDIA's own benchmark.

That is roughly 2× better cost per token, for less money per hour, on the same cloud. If you are running inference on a3-highgpu-8g today, that is a one-line change to your instance type and it is very likely the highest-return hour of work available to you at this layer.

Check the equivalent on your own cloud before acting — availability and quota differ, the benchmark is one model at one shape, and your workload is not Llama2 70B at ISL 2K. But the direction is not subtle, and it is a direct consequence of the physics above rather than a vendor claim.

What this chapter is not saying

Compute is not irrelevant. Prefill is genuinely compute-bound, training is compute-bound, and at very large batch sizes decode crosses the roofline into compute-bound too. Models with heavy attention at long context shift the arithmetic. Mixture-of-experts models read only active parameters, which changes the weight-bytes term substantially.

The claim is narrower and it holds: for single-stream and modest-batch decode — the shape most production inference actually has — memory bandwidth sets the ceiling, and buying compute you cannot feed is the most common expensive mistake at this layer.