The non-NVIDIA pitch is always the same: same work, less money. It is usually supported by an hourly rate, and the hourly rate is the one number in this comparison that cannot answer the question.
This chapter uses one vendor's own price feed, one region, one day — AWS on-demand in us-east-1 — so the NVIDIA and non-NVIDIA numbers cannot differ because of who is quoting them. Then it applies the metric this book established as the product: dollars per TB/s of memory bandwidth, because decode is bandwidth-bound.
Two orderings that disagree
| Instance | Silicon | $/hour | Aggregate HBM bandwidth | $/hr per TB/s |
|---|---|---|---|---|
| inf2.48xlarge | 12 × Inferentia2 | $12.98127 | 9.8 TB/s | $1.3246 |
| TPU v5e (1 chip) | Google TPU v5e | $1.20 | 0.859 TB/s | $1.3970 |
| p5.48xlarge | 8 × H100 SXM | $55.04 | 26.8 TB/s | $2.0537 |
| trn1.32xlarge | 16 × Trainium1 | $21.50 | 9.8 TB/s | $2.1939 |
Sorted by sticker price the order is TPU, Inferentia2, Trainium, H100 — cheapest to dearest. Sorted by the thing decode consumes, Trainium moves from second-cheapest to most expensive on the list.
The Trainium inversion
A trn1.32xlarge is 60.9% below a p5.48xlarge per hour: $21.50 against $55.04. That is the number in the pitch, and it is true.
Per TB/s of memory bandwidth it is 6.8% more expensive than the H100 — $2.1939 against $2.0537, measured against the H100's rate. Turned around, the H100 is 6.4% cheaper per TB/s than Trainium, measured against Trainium's.
Nothing dishonest has happened. A trn1.32xlarge simply is a smaller machine: 9.8 TB/s of aggregate bandwidth against the p5's 26.8. The 60.9% is a size discount, not a value discount — you are buying 37% of the bandwidth for 39% of the price. Run the same token volume through it and the saving disappears, because you need proportionally more of them.
This is the single most common error in accelerator selection, and it survives because the hourly rate is the number on the page and the bandwidth is three clicks away.
The part that actually is cheaper
Look one row up. Inferentia2 is 35.5% cheaper per TB/s than the H100 — $1.3246 against $2.0537, measured against the H100's rate; the H100 is 55.0% more measured against Inferentia2's. Same vendor. Same feed. Same day.
So AWS does sell a genuinely better bandwidth deal than an H100 — and it is the inference part, not the training part. Trainium is the one with the marketing behind it and it is the one that loses on this metric; Inferentia2 wins by a third and is barely mentioned.
If you are serving tokens rather than training, that is the row to look at.
TPU v5e, and a units trap that would have flipped it
TPU v5e at $1.20 per chip-hour comes to $1.3970/TB/s — 32.0% below the H100, a real advantage of the same order as Inferentia2's.
But notice how the bandwidth was converted. Google publishes 800 GiBps per chip. NVIDIA publishes 3.35 TB/s. Those are different units: 800 GiB/s is 0.858993 TB/s, not 0.8. Treating the TPU figure as 0.8 TB/s understates it by 6.87% — which is larger than the entire Trainium-versus-H100 gap this chapter just measured. A silent GiB/GB conflation would have been big enough to reorder the table.
Google's other rungs price at $2.70 per chip-hour (Trillium) and $12.00 (Ironwood), but this book does not have per-chip bandwidth specifications for those from a primary source, so they are quoted as prices only and excluded from the normalised table.
A third ordering: capacity
Because VRAM sizing showed capacity and speed have different winners, here is the same set on dollars per GB of HBM per hour:
| HBM | $/GB-hour | |
|---|---|---|
| inf2.48xlarge | 384 GB | $0.0338 |
| Trainium1 (trn1.32xlarge) | 512 GB | $0.0420 |
| TPU v5e (1 chip) | 16 GB | $0.0750 |
| p5.48xlarge | 640 GB | $0.0860 |
Trainium beats the H100 by 2.05× on capacity while losing to it on bandwidth. Both are true and they are not in conflict — they answer different questions. If your constraint is fitting a large model at all, Trainium's 512 GB at $0.0420/GB-hour is the better buy. If your constraint is decode speed, it is the worst thing in this chapter.
That is three orderings from four accelerators, and picking the wrong one is how a migration gets approved on a number that does not apply to the workload.
AMD
The MI300X specification is 192 GB of HBM3 at 5.3 TB/s — more memory than an H100 SXM and more bandwidth, on paper the strongest single-chip position in this chapter.
I could not source a like-for-like price for it and have not estimated one. It is not in the AWS on-demand feed used above, and pricing it from a different vendor on a different day would break the one control that makes this chapter's comparison mean anything. The specification is quoted; the economics are not, and a table with three sourced rows and one guessed row would be worse than a table with three.
The porting cost, unquantified and real
Every option here requires a different compiler stack — AWS Neuron, XLA, ROCm — and none of them is a drop-in for a CUDA-targeted serving stack. This book has no defensible number for what that migration costs, and I am not going to invent one; engineering-time estimates dressed as measurements are exactly what the rest of this book refuses to do.
What can be said precisely is where the number has to come from: it is your team's rate times the time to get your model, at your precision, through their compiler and passing your evals — and the honest way to find it is a two-week spike on one model before anything is committed, not a spreadsheet.
The bandwidth arithmetic above tells you whether that spike is even worth booking. For Trainium on a decode-bound workload, it is not: you would be paying migration cost to arrive at a worse rate.
The diagnostic
- Never compare accelerators on the hourly rate. It encodes machine size, not value. The four instances here span 46× in price and the ranking inverts under normalisation.
- Normalise on the thing your workload consumes. Decode-bound serving: dollars per TB/s. Fitting a large model: dollars per GB. They will not agree.
- Take prices from one vendor's feed on one day where you can. AWS quotes H100, Trainium and Inferentia2 in the same file — that is a controlled comparison you get for free, and almost nobody uses it.
- Check the units before the conclusion. GiB/s and GB/s differ by 7.4%, enough to reorder this table on its own.
- Look past the chip with the marketing. On this data the inference-specific part beats the training-specific part by 40% per TB/s, from the same vendor.
- Price the migration as a spike, not an estimate. Two weeks on one model, then decide.
What this chapter is not saying
It is not saying Trainium is a bad chip. It is saying the number it is sold on does not describe decode, and that the same vendor sells something better for that job.
It is not saying these four are interchangeable. They differ in compiler maturity, model coverage and operational familiarity in ways this chapter does not measure, and every one of those differences is a cost that lands after the invoice comparison is over.
It is saying that the accelerator comparison almost everyone runs — hourly rate, two vendors, one screenshot — reliably produces the wrong ranking, and that the correction takes one division.