Waqar Uddin

There are three prices for the same token

November 9, 2026 (1w ago)3 views

Disclosure: I work at Jazz as Principal Evangelist Cloud & AI. This article is written as personal industry analysis. Views are my own.

Harry Stebbings published his notes from a conversation with Arthur Mensch of Mistral last week. One line deserves more attention than it got. In his summary, on-demand inference behaves like a high-margin commodity market where pricing matters a lot and demand comes from non-committed customers. Mature customers, the notes say, want service level agreements, reserved physical capacity, and guarantees on latency and generation speed. And the more precisely a customer can describe the shape of its demand, the better Mistral can price it.

Read that as a buyer. It says the price you pay for intelligence is not a function of the model. It is a function of how well you can forecast yourself.

The same GPU hour already sells at three different prices, and the spread is wide enough to decide a business case.

The three prices

Track the data center GPU market and the structure is visible. Across 27 providers, the H100 rents anywhere from $0.67 to $5.98 per GPU hour on spot, up to $11.06 on demand, and $1.79 to $7.67 reserved. The cheapest reserved rate is roughly a quarter of the most expensive on-demand rate for the identical card. One tracker that pairs each provider's committed rate against its own on-demand rate puts the median one-year discount at 16% and the median three-year discount at 41%. AWS lists the H100 at $6.88 on demand and $2.97 reserved. That is 57% off for a term commitment.

The token market has the same shape, one layer up. Anthropic and OpenAI both price batch processing, the same model, the same tokens, the same quality, at half price if you can wait up to a day. Anthropic prices a cached read at a tenth of base input tokens, and a five-minute cache write at 1.25 times. Some of their newest models price cache reads at a twentieth of base. The discount multipliers stack by design.

Nothing about the model differs across these prices. What differs is who carries the risk of idle capacity and unpredictable demand. Spot shifts utilisation risk to you through eviction. Reserved shifts it away from the seller, so they pay you for taking it. On demand, the seller carries it and charges you for the option value. When Mensch calls on-demand inference high margin, that is not a contradiction of the word commodity. It is the definition of selling insurance to buyers who do not know their own demand.

Why this is a procurement problem now

Twelve months ago an enterprise bought a chatbot pilot, spent a few thousand dollars on tokens, and nobody needed a term structure. The pilot either died or it scaled. When it scales, the notes from the Mistral conversation describe exactly what happens: a customer arrives needing thousands of GPUs, not eight, and the provider asks what shape of demand this is, because the answer decides whether it gets a commodity price or an infrastructure contract.

Most buyers cannot answer. They do not log tokens by task. They do not know their peak-to-average ratio, their context length distribution, or how much of their prompt is a repeated prefix that a cache should absorb. So they sit in the on-demand tier, at the highest per-unit price, with no service level agreement, and they experience the resulting bill as a complaint about the vendor.

The bill is real. I keep a ledger of every agent run in my own factory, model by model, tokens in and out, and the difference between the runs that hit the cache and the ones that re-send a full context every time is routinely a factor of five on the same workload. That gap is not engineering skill. It is accounting.

Four numbers before you negotiate

If you buy inference and cannot yet produce these, that is the work, and none of it is hard:

  1. Tokens per month, split input and output, split by task class. Classification, extraction, drafting and reasoning have different shapes and belong to different tiers.
  2. Peak concurrency and the peak-to-average ratio. This is the number that decides whether you need reserved throughput or can ride spot for the surge.
  3. Context length distribution and repeated prefix share. If 80% of your prompt is the same system block and document set every call, you are buying prefill you already paid for. Cache it.
  4. Latency tolerance per task. Anything the user does not wait on belongs in the batch tier at half price. In most workloads that is half the volume, and it is the cheapest 50% you will ever cut.

With those four numbers you can walk into a negotiation, or a build decision, and buy down the curve: batch the tolerant volume, cache the repeated prefix, reserve the floor, spot the surge, and own the box for the flat part of the load if the crossover arithmetic says so. Without them, you are a non-committed customer by definition, and the market is priced accordingly.

The Pakistan version of the uncertainty tax

We pay this tax twice. The on-demand tier bills in dollars, so the uncertainty premium sits on top of an exchange rate premium. And our buyers are structurally non-committed: volumes are small, forecasting discipline is thin, and nobody wants to sign a term for capacity they cannot prove they need. So we buy the most expensive tier of a dollar-denominated commodity and call it the cost of AI.

The sovereignty conversation here keeps gravitating to hardware, a box, a data centre, a national GPU cluster. Mensch's framing suggests the contract matters as much as the silicon. The opposite of dependent is not owned. It is committed, with a service level agreement, a published per-token price, reserved capacity, and data residency, from a counterparty you have actually underwritten. A country or a bank that cannot describe its demand shape cannot buy that, no matter how many GPUs it imports. That is the part of sovereign AI nobody sells, because it cannot be invoiced as a line item.

Certain, signal, unresolved

Certain, because the price pages say so: the same H100 hour sells from $0.67 on spot to $11.06 on demand. Median one-year commitments cut 16%, three-year cut 41%. Batch is half price at both major labs. Cached reads are a tenth of base input, and cheaper on the newest models.

Signal, and this is my reading: the inference market has finished its commodity phase and started its term structure phase. Margin is migrating from the model to the risk transfer, and the buyers who win are the ones who can describe their demand precisely enough to be sold something cheaper than an option.

Unresolved: whether enough buyers will ever do the accounting. The easy outcome is that resellers aggregate the uncertainty and keep the spread, and enterprises keep buying packaged on-demand inference at a markup. That is a business model, and it will work right up until one customer at the table has its four numbers.

Related reading: The token factory maths nobody runs for the crossover arithmetic between renting and owning, The agent moves onto your desk for the local end of the curve, and The reset wall is a walk-away now for what happens to sellers when committed buyers disappear.

Sources: Harry Stebbings' notes from his conversation with Arthur Mensch, Cloud GPUs H100 rental price index, GPU Economy committed-rate medians, Anthropic batch processing, Anthropic prompt caching multipliers, and OpenAI Batch API.