Every chapter so far has treated one machine as the unit. Interconnect is what happens when the model does not fit in one, and it introduces a discontinuity that nothing else in this book has: a step change in bandwidth that occurs at a physical boundary, not a gradual one.
Two things need separating, because the industry conflates them. The fabric is the wiring between accelerators, and it is priced inside the instance. Data transfer is what you pay to move bytes between zones, regions, or out to the internet, and it is billed separately. This chapter can price the first precisely and cannot price the second at all — see the end.
The cliff
Take a p5.48xlarge: eight H100 SXM in one chassis, 3,200 Gbps of Elastic Fabric Adapter, $55.04/hour on-demand. Three bandwidth numbers describe the same eight GPUs.
| Path | Per GPU | Source |
|---|---|---|
| HBM — GPU to its own memory | 3,350 GB/s | NVIDIA H100 SXM spec |
| NVLink — GPU to GPU in the box | 900 GB/s | NVIDIA H100 SXM spec |
| EFA — GPU to a GPU in another box | 50 GB/s | 3,200 Gbps ÷ 8 bits ÷ 8 GPUs |
Crossing from a GPU's own memory to its neighbour's costs 3.72×. Crossing from its neighbour to a GPU one metre away in the next chassis costs another 18×. End to end, local memory is 67× the bandwidth of the inter-node path on the same machine.
That 18× is not a bad-network number. 3,200 Gbps is the fastest fabric in the AWS fleet by a wide margin — the previous generation, p4d.24xlarge, offered 400 Gbps, so this is an 8× generational jump in exactly the bottleneck everyone complained about. It is still an 18× cliff, because NVLink moved too.
This is why topology is a cost question and not a networking question. Tensor parallelism splits a single matrix multiply across GPUs and exchanges activations on every layer; run it inside the NVLink domain and the exchange is nearly free, run it across the cliff and the exchange is the workload. The placement decision is worth 18× on the communication term, and no amount of instance-shopping recovers it.
The domain is the number that matters
NVIDIA publishes a switch generation table, and the interesting column is not bandwidth.
| NVLink 4 (Hopper) | NVLink 5 (Blackwell) | |
|---|---|---|
| GPU domain | 8 | 8 or 72 |
| GPU-to-GPU | 900 GB/s | 1,800 GB/s |
| Total aggregate | 7.2 TB/s | 130 TB/s (NVL72) |
Per-GPU bandwidth doubled. The domain grew 9×. Multiply those and you get the 18.06× jump in aggregate — the domain contributed twice what the link speed did.
Two footnotes on that table, both worth the seconds they take. Hopper's aggregate is exact: 8 × 900 GB/s is 7.2 TB/s to the digit. Blackwell's is rounded: 72 × 1,800 GB/s is 129.6 TB/s, published as 130, 0.31% generous. Harmless here, and a reminder that headline aggregates are marketing arithmetic even when they are nearly right.
The practical reading: a model that needs more than eight GPUs of tensor parallelism falls off the cliff on Hopper and does not on an NVL72. That is a change in which models are economically servable, not a percentage improvement. This book has no price for an NVL72 instance — none appears in the AWS on-demand feed used throughout — so the economics of that change are not quantified here.
What a fabric actually costs
AWS sells network-upgraded variants of many instance families, marked with an n. Two of those pairs are close enough to controlled experiments to measure the fabric directly.
trn1n.32xlarge against trn1.32xlarge is as clean as this book has found. Identical processor generation, identical 128 vCPUs, identical 512 GiB of RAM, identical 16 Trainium accelerators with 512 GiB of accelerator memory, and identical EBS bandwidth — 80,000 Mbps, 10,000 MB/s, 260,000 IOPS on both. The single difference AWS documents is network cards: 8 versus 16, for 800 versus 1,600 Gbps.
That doubling costs 15.26% — $24.78 against $21.50.
Now the same fact three ways, because only one of them is useful:
| trn1.32xlarge | trn1n.32xlarge | |
|---|---|---|
| $/hour | $21.50 | $24.78 |
| Network | 800 Gbps | 1,600 Gbps |
| $/hr per Gbps | $0.026875 | $0.01549 |
The upgraded machine is 42.37% cheaper per gigabit than the base one. And the marginal gigabit — $3.28 divided by the 800 Gbps it buys — costs $0.0041/Gbps-hour, 6.55× less than the average gigabit on the base machine.
Fabric priced at the margin is cheap. Fabric priced at the average looks expensive. Almost every "we can't afford the networking tier" conversation is conducted in averages.
The tier that decides whether you can run the job at all
The m5 pair adds something the Trainium pair cannot show, because Trainium has EFA on both rungs.
| m5.24xlarge | m5n.24xlarge | |
|---|---|---|
| vCPUs / memory | 96 / 384 GiB | 96 / 384 GiB |
| EBS bandwidth | 19,000 Mbps | 19,000 Mbps |
| Network | 25 Gbps | 100 Gbps |
| EFA | No | Yes |
| $/hour | $4.608 | $5.712 |
Same core count, same memory, same storage bandwidth — the CPU stepping differs (Xeon 8175 against 8259) and nothing else does. $1.104/hour, a 23.96% premium, buys 4× the network and the right to use EFA at all.
Per gigabit the upgraded instance is 69.01% cheaper, and the marginal gigabit costs $0.01472 — one twelfth of the average gigabit on the base machine (12.52×, precisely).
The EFA row is the one that matters. A tightly-coupled distributed job on the base instance is not slower; it is a different and much worse thing, because it is running collectives over ordinary TCP. Roughly a quarter more per hour is the price of admission to the class of workloads this chapter is about. Paying it and then under-using the fabric is a cheap mistake. Not paying it and discovering the job is communication-bound is an expensive one.
The premium is a function of how expensive the machine already was
Across the same-size n pairs in the feed, the network upgrade is a wildly inconsistent share of the bill:
| Pair | Network | Premium | Delta as share of upgraded bill |
|---|---|---|---|
| trn1 → trn1n (32xlarge) | 800 → 1,600 Gbps | 15.26% | 13.24% |
| m5 → m5n (24xlarge) | 25 → 100 Gbps | 23.96% | 19.33% |
| c6i → c6in (32xlarge) | 50 → 200 Gbps | 33.41% | 25.04% |
| r6i → r6in (32xlarge) | 50 → 200 Gbps | 38.36% | 27.72% |
| m6i → m6in (32xlarge) | 50 → 200 Gbps | 45.03% | 31.05% |
| c7g → c7gn (16xlarge) | 30 → 200 Gbps | 72.14% | 41.91% |
On a Graviton compute instance, 41.91% of what you pay for the networked variant is the network. On a Trainium instance, 13.24%. The fabric costs roughly what it costs; the machine underneath it does not. That ordering is the reason accelerator instances ship with enormous fabrics as standard and CPU instances make you choose.
Only the top two rows are controlled comparisons. The four 6i/7g rows also buy 2.5× EBS bandwidth, and c7gn additionally changes the processor to Graviton3E — so those premiums bundle storage and silicon with the network and are directional, not measurements. Stated rather than silently averaged in.
The pair you must check before you trust the delta
Name-matching instance types is how this analysis goes wrong. In the same feed, on the same day:
- c5.metal — $4.08/hour, 96 vCPUs, 25 Gbps
- c5n.metal — $3.888/hour, 72 vCPUs, 100 Gbps
Read as a pair, the network variant is 4.71% cheaper — an apparently negative network premium, which would be a remarkable finding if it were one. It is not. c5n.metal has 24 fewer vCPUs. You are looking at a smaller machine that happens to have a bigger NIC.
The nearest honest c5 pairing is c5.18xlarge against c5n.18xlarge — 72 vCPUs each, same Xeon 8124M, same EBS — at +27.06%. Even that is not clean, because c5n carries 192 GiB against 144.
Confirm vCPUs, memory, and EBS bandwidth match before attributing a price delta to the network. It takes one lookup and it is the difference between a measurement and a coincidence.
The part with no fabric, on purpose
inf2.48xlarge — the instance that won the bandwidth-per-dollar comparison — has 100 Gbps and no EFA support at all. AWS did not forget. Inference on a model that fits in one instance's accelerator memory never crosses the cliff, so the fabric would be dead weight in the price.
That is the clean statement of the whole topic. The interconnect bill is a function of whether your model fits, which is the question VRAM sizing answers and quantization moves. Halving the weights to fit inside one NVLink domain does not just save memory — it removes an 18× bandwidth cliff from the critical path, and that saving never appears in a quantization benchmark.
The diagnostic
- Find the domain boundary before choosing a parallelism strategy. Eight GPUs on Hopper. Everything past it pays 18×.
- Price fabric at the margin, not the average. The extra 800 Gbps on a trn1n costs 6.55× less per gigabit than the network already in the base machine.
- Treat EFA as a capability gate, not a speed feature. The m5 → m5n step is 23.96%; without it, a tightly-coupled job is running collectives over TCP.
- Check vCPUs, memory and EBS before believing an
n-variant delta. c5.metal versus c5n.metal reads as a 4.71% discount and is a 24-vCPU difference. - Expect the premium to scale inversely with machine cost. 13.24% of a Trainium bill, 41.91% of a Graviton one.
- Recompute the vendor's aggregate. 8 × 900 GB/s is exactly 7.2 TB/s; 72 × 1,800 GB/s is 129.6, published as 130.
What this chapter cannot tell you
No InfiniBand hardware or switch price appears here. Fabric silicon is sold through system vendors on quote, not published rate cards, and the AWS figures above are the price of access to a fabric, not the cost of building one. Anyone comparing cloud fabric to owned fabric needs the second number, and this book does not have it.
No data-transfer pricing appears here either. The metered-unit map this book uses throughout contains no data-transfer keys, and AWS's transfer and VPC pricing pages render client-side. Inter-AZ and inter-region movement is a real and sometimes dominant line item, and it is genuinely unsourced in this book rather than omitted as unimportant.
No collective-communication overhead figure appears here. What fraction of your step time is spent in all-reduce depends on model shape, batch size, and framework, and the honest way to get it is a profiler on your own job — the same answer utilisation got, for the same reason.
What is precisely established is narrower and still useful: the cliff is 18× on the best fabric AWS sells, the domain is eight GPUs on Hopper, and the fabric that crosses it costs between 13% and 42% of the machine depending entirely on how expensive the machine already was.