Almost every conversation about AI cost starts in the wrong place. Someone opens a pricing page, compares two dollar figures per million tokens, picks the smaller one, and calls it an optimization. Then the bill goes up anyway, and nobody can say why.
The reason is that price per token is one of four terms in the bill, and usually not the one moving it.
The actual equation
Your spend on a language model is:
cost = (input tokens x input rate) + (output tokens x output rate), summed over every call
Four things you can change, then:
- The input rate and output rate — which model you call.
- The input token count — how much context you send.
- The output token count — how much text you ask it to generate.
- The number of calls — how many times the whole thing happens.
Switching models moves term 1. That is the lever everyone reaches for first, because it is the only one visible on a pricing page. It is frequently the weakest one available, because the other three are set by your architecture, and architecture usually has more slack in it than procurement does.
A concrete version of the problem: if you send a 40,000-token system prompt on every request to answer a question that needs 400 tokens of context, you are paying a hundred times over for the privilege of not having built retrieval. No model swap fixes that. Moving to a model half the price leaves you paying fifty times over.
A token is not a word
A token is a chunk of text the model's tokenizer produces — usually a common word, a word fragment, or a punctuation mark. For ordinary English prose, a rough working figure is about 0.75 words per token, which is to say roughly 4 characters. That ratio is a planning aid, not a fact about your workload, and it degrades in specific and expensive ways:
- Code tokenizes worse than prose. Whitespace, braces, and identifiers fragment.
- Non-English text frequently tokenizes far worse, sometimes several times worse, because the tokenizer's vocabulary was fit predominantly to English. A Hindi or Thai sentence can cost multiples of its English translation for the same meaning.
- JSON carries a structural tax. Every key, brace, and quote is billable. A response schema with verbose field names is a recurring charge you pay on every single call.
- Base64 and long identifiers fragment into near-worthless tokens.
If your product serves a non-English market or moves structured data, your effective cost per unit of meaning is worse than the pricing page implies, and the pricing page will never tell you that. Measure your own ratio. It is one line of instrumentation and it changes how you read every other number.
Why the two rates differ
Look at any provider's pricing and you will see output priced above input — commonly several times above. This is not a margin decision. It follows from how inference actually runs, and it is the single most useful thing to understand about model economics, so it gets its own chapter.
The short version: reading your prompt and writing the answer are different computations with different hardware bottlenecks. One parallelizes and one does not.
What this means before you optimize anything
Three habits, in order:
Measure tokens, not dollars. Dollars mix rate changes with volume changes, so a dollar figure cannot tell you whether your bill moved because you got more traffic or because someone lengthened a prompt. Log input and output token counts per call, tagged by feature. Everything downstream depends on having this.
Get cost per unit of work, not cost per month. "We spend $18,000 a month on inference" is not actionable. "A document review costs us $0.14, of which $0.11 is one summarization step" tells you exactly where to go. The denominator matters more than the numerator.
Find your top three call sites before touching anything. Model spend is reliably concentrated. A small number of code paths generate most of the tokens, and until you know which, any change you make is a guess. This is the same shape as every performance problem you have ever debugged, and it responds to the same discipline: profile first.
The honest caveat
Published list prices are a starting point and nothing more. They are not your negotiated rate at volume, not your effective rate once caching and batch discounts apply, and not a measurement of what any task costs you. The live price index on this site is useful for shortlisting a model and for sanity-checking an invoice. It cannot tell you what you are spending. Only your own usage data can do that.
That gap — between what the pricing page says and what the bill says — is where nearly all the recoverable money sits.