Managed inference endpoints

Between renting a GPU by the hour and calling a frontier API sits a third thing: someone else runs an open-weights model, you call it, you pay per token. Bedrock, Vertex, Azure AI Foundry, Together, Fireworks, Groq, DeepInfra all sell this.

This layer is unusually easy to price honestly, because the product is genuinely identical. When you buy Claude from Anthropic and GPT from OpenAI, you are comparing different models and the comparison is partly aesthetic. When you buy Llama 3.3 70B, every vendor is serving the same weights. Any price difference is the vendor's margin, their serving efficiency, and what else they bundle — nothing about the model.

So it is a controlled experiment, and the result is worth the page.

One model, four vendors

Published rates on 2026-08-05, US region, on-demand, per million tokens:

Blended assumes 750k input and 250k output per million tokens — a typical RAG-ish shape. Change the mix and the ordering holds; only the gaps move.

A 6.7× spread on identical weights. That is wider than the 4.7× spread on H100-hours from what an H100-hour costs, and it requires no migration, no new hardware, and no serving expertise to capture. It is a base-URL change.

Three things this table says

AWS and Google charge exactly the same price. Not approximately — $0.72 input and $0.72 output on both, to the cent, on both sides of the meter. Two companies that agree on nothing else have landed on an identical number for the same open model. Whatever produced that, it is not independent cost-plus pricing. Treat $0.72 as the hyperscaler list rate for Llama-class inference, and treat "we compared Bedrock and Vertex and they were close" as evidence of nothing.

Together AI is more expensive than both hyperscalers. This is the finding that should change how you shop. The reflex — specialists are cheaper than clouds, that is the whole point of specialists — is simply false here. Together lists $1.04, which is 44% above Bedrock and Vertex. A team that moved off Bedrock to Together to save money on this model would have increased its bill by 44% while adding a vendor.

The cheap option is 4.6× below the hyperscalers, not 20% below. DeepInfra's $0.10 input is a different order of magnitude, not a discount. When a gap is that wide it usually means the vendors are not selling the same thing — and the rest of this chapter is about establishing whether they are.

Note also that input and output are priced identically at three of the four vendors. That is unusual, and it means the output premium that dominates frontier-API economics mostly disappears here. DeepInfra is the exception, at 3.2× output over input.

Where the money actually goes: the self-host crossover

The honest way to test whether a token price is good is to price the alternative: rent the GPU and serve it yourself.

An H100 on RunPod is $2.99/GPU-hour. To beat a token price by self-hosting, your sustained throughput has to clear:

tokens per second = $/hour ÷ (price per 1M tokens × 3600) × 1,000,000

Run it against both ends of the table:

That is the whole strategic picture in two rows.

Against the hyperscaler endpoints, self-hosting wins at moderate scale. 1,150 tok/s on a single H100 serving a 70B model is a reasonable target for a well-batched vLLM deployment — which is exactly what continuous batching buys you. If you are spending real money on Bedrock or Vertex for an open model, the arithmetic says look at your own endpoint.

Against DeepInfra, self-hosting almost certainly loses. 5,360 tok/s sustained on one H100 is at the edge of what the hardware does for a 70B model, and you would need to hold it at high utilisation around the clock to realise it. DeepInfra is selling below what most teams can build. Buying it is not laziness; it is the correct call.

The general rule this produces: a managed endpoint is worth paying for exactly when its price is below your achievable self-host cost, and the gap between vendors at this layer is wide enough that the answer flips depending on which vendor you priced. Teams that conclude "managed inference is expensive, we should self-host" have usually priced only the expensive vendors.

What the cheap price is not buying you

A 4.6× gap survives scrutiny only if you scrutinise it. Before moving spend:

• Throughput and latency are not in the price. Same weights does not mean same tokens-per-second or same time-to-first-token. Vendors differ in serving stack, batching policy, quantization, and how hard they pack their GPUs. A cheap endpoint that is 3× slower may cost more per unit of work. Measure yours; the price table cannot tell you this.
• Quantization is often the explanation. Several vendors serve FP8 or INT8 versions of "the same" model. That is frequently the right trade — it is a bandwidth optimisation that makes serving genuinely cheaper — but it is not bit-identical to the reference weights, and for evaluation-sensitive work you need to know which you are getting.
• Capacity and rate limits. Hyperscaler endpoints come with quota you can raise through an account team. Independents vary from generous to abrupt.
• Compliance and data handling. Residency, retention, a signed BAA, and whether your tokens train anything. This narrows the eligible list faster than any other factor, and it is the honest reason many regulated teams pay the $0.72.
• Everything already in your cloud. If your data, VPC, and IAM live in AWS, Bedrock is a private-link call and DeepInfra is egress plus a new vendor review. That is real cost, though rarely 4.6× of it.
• Model longevity. Independents deprecate aggressively. Hyperscalers keep old versions available longer, which matters if you have pinned a version behind an eval suite.

None of these make $1.04 correct when $0.155 exists for the same weights. They do mean the comparison is total cost of the workload, not the headline rate — and a 6.7× gap is wide enough that it usually survives the accounting, which is exactly why the accounting is worth doing.

The diagnostic

1. For every open-weights model you serve, what are you paying per 1M tokens, and what are the other three vendors charging for the identical model today? This is a fifteen-minute exercise and it is the highest-return fifteen minutes at this layer.
2. Is your blended rate above ~$0.72? If so you are paying above the hyperscaler list rate for a commodity, and you should know why.
3. What is your sustained tokens/second and utilisation? Put it against the crossover table. Below ~1,150 tok/s of real demand, self-hosting a 70B model on a dedicated H100 is hard to justify.
4. Are you comparing quantizations? If you have not checked, you are possibly comparing an FP8 endpoint against a BF16 one and calling the difference vendor margin.
5. Have you priced batch? Bedrock's batch tier for this model is $0.36 input — half of on-demand. Anything asynchronous should be on it.

Question 5 is the one most often missed. Half off, for work that was never latency-sensitive, with no vendor change.

What this chapter is not saying

It is not saying use the cheapest row. It is saying the spread is 6.7×, it is invisible unless you look, and the intuition most teams use to shop this layer — specialists undercut clouds — produced the wrong answer for one of the four vendors here.

It is also not a claim about frontier models. Claude, GPT, and Gemini are not commodities and are not priced like these; you cannot arbitrage them across vendors because only one vendor sells each. This chapter is specifically about the open-weights layer, where the same artefact has four prices.

And prices at this layer move faster than anywhere else in this book. The 2026-08-05 numbers will be stale. The method — pick your model, price it at four vendors, compute the self-host crossover — will not be.