VRAM sizing

The accelerator ladder ended on a gate it refused to compute: 48 GB is 48 GB. This chapter computes it.

VRAM holds two things with completely different behaviour. Weights are a fixed tax — paid once, identical whether you serve one request or a thousand. KV cache is the variable — it scales with concurrency × context and is the term that actually decides how many users a GPU can hold. Almost every sizing conversation gets the first right and never does the second.

Weights: the fixed tax

Parameter counts read from the safetensors metadata of four open-weights models, not from their names:

Two things worth pausing on.

Mixtral "8x7B" is 46.7 billion parameters, not 56. The attention layers are shared across experts, so the name overstates it — and understates the problem, because all 46.7B must be resident while only two of eight experts run per token. You pay for the whole mixture in VRAM and use a quarter of it per forward pass.

Qwen2.5-72B at FP16 is 135.43 GiB — larger than every GPU on the ladder except the B200 and B300. At FP8 it drops to 67.71 GiB and fits an 80 GB card with room to spare. That single choice moves the model across the feasibility line.

KV cache: the term that scales

Per token, the cache is 2 × layers × kv_heads × head_dim × bytes — the 2 being one K and one V. For Qwen2.5-72B, from its config.json: 80 layers, 8 key-value heads, head dimension 128, at 2 bytes:

2 × 80 × 8 × 128 × 2 = 327,680 bytes = 320 KiB per token.

One user, one conversation, at Qwen2.5-72B's full documented context, costs 10 GiB. Not the model — the conversation. Ten of those is 100 GiB of cache alone, more than an H100 holds with no weights in it at all.

The config field that makes this affordable

Qwen2.5-72B has 64 attention heads and 8 key-value heads. That ratio — grouped-query attention, 8:1 — is the whole reason the number above is survivable.

Without it, with one KV head per attention head, the same formula gives 2,560 KiB per token. At 32K context that is 80 GiB for a single request — an entire H100 SXM, serving one user.

num_key_value_heads is a line in a JSON file that almost nobody reads and that changes your serving cost by up to 8×. It is the highest-leverage number in the config.

What actually fits, and what it costs

Qwen2.5-72B at FP8 — 67.71 GiB of weights — against the ladder priced in the previous chapter. The free column is what remains for cache; n is concurrent requests; the last figure is dollars per concurrent-request-hour:

An H100 SXM running this model at full context serves exactly one user. At $2.99/hour.

The non-linearity, and why it overturns the last chapter

Compare the H100 SXM and H200 rows. The H200 has 1.76× the VRAM and costs 1.47× more per hour — both modest. But after the same 67.71 GiB of weights come off the top, it has 5.96× the free space, so it serves 58 concurrent requests against 9 — 6.4× the concurrency.

Per concurrent request, the H200 is 4.39× cheaper than the H100 SXM.

That is a direct correction to the accelerator ladder, which ranked the H200 worse than several cheaper cards on dollars per gigabyte. Both tables are arithmetically right; the $/GB one asks the wrong question. Gigabytes occupied by weights produce zero throughput. Only the remainder serves users, and the remainder grows non-linearly with capacity because the weights term is constant.

The same mechanism makes the B300 — the most expensive GPU on the ladder — 7.91× cheaper per concurrent request than the H100 SXM at 4K context.

The honest caveat: fitting is not serving

The A100 SXM and H100 SXM fit identically here — same 80 GB, same 9 concurrent requests — and the A100 costs half as much. That does not make it half the price for the same service.

Memory bandwidth is the product established that decode speed tracks bandwidth, and the H100 SXM's 3,350 GB/s is 1.64× the A100 SXM's 2,039 GB/s. Both hold nine conversations; the H100 answers them faster.

Dollars per concurrent-request-hour measures capacity, not speed. A table that ranked cards on capacity alone would mislead exactly as badly as the $/GB table it corrects. Use it to answer "how many users fit"; use bandwidth to answer "how fast do they get tokens"; buy on whichever is binding.

The diagnostic

1. Compute weights first, at the precision you will actually serve. FP16 → FP8 halves it and often decides feasibility outright.
2. Compute 2 × layers × kv_heads × head_dim × bytes from the model's real config.json. Do not use the parameter count as a proxy — a 7B and a 72B here differ by only 5.7× in cache per token while differing 9.5× in weights.
3. Check num_key_value_heads against num_attention_heads. A model without GQA can cost 8× more per token of context than one with it, at the same parameter count.
4. Multiply by your real context and your real concurrency. Ten users at 32K on this model is 100 GiB — a bigger number than the model.
5. Subtract weights from VRAM before ranking anything. The free remainder is the number that serves users, and it does not scale with the label on the box.
6. Then check bandwidth separately. Capacity and speed are different constraints with different winners.

What this chapter is not saying

It is not saying buy B300s. It is saying that the standard sizing question — "does the model fit?" — is the easy half, and the half that is usually wrong is the one nobody computes.

A model that fits with 12 GiB to spare fits. It also serves one user at long context, which for most products is the same as not fitting at all.