Waqar Uddin

The token factory maths nobody runs

October 24, 2026 (14h ago)3 views

Disclosure: I work at Jazz as Principal Evangelist Cloud & AI. This article is written as personal industry analysis. Views are my own.

One number broke my sense of what inference costs

Earlier this month a post about DeepSeek's inference deployment kept crossing my feed, and one line in it stopped me. A single cluster, it said, serves 155 trillion tokens a day at overall revenue of $38 per GPU-hour.

I did the division on a napkin. 155 trillion tokens split across 128 GPUs over 24 hours is about 14 million tokens per GPU-second. At $38 an hour, that works out to roughly $0.00075 per million tokens. Three quarters of a tenth of a cent for a million tokens.

If that were a per-request price, inference would be a rounding error, and every business plan built on GPU rental would be worthless. It is not a per-request price. It is a fleet average, and the gap between those two readings is where most of the commentary about this repository has gone wrong.

Where the claim actually lives

The number is real, but it sits in one document: the inference deployment notes that ship with the repository. The wording there is precise: a single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour, at valley pricing.

Two other numbers get quoted alongside it and are not in the text at all. The 25 to 35 percent throughput gain from running at 64 GPUs of expert parallelism instead of 8, and the 60 to 90 thousand tokens per second per GPU figure, appear only inside a Pareto chart in that document. I searched the repository's text files and neither figure occurs in any of them. That matters, because people are citing a picture as if it were a benchmark.

What the document does say is more useful. The smallest efficient deployment unit is 64 prefill GPUs and 64 decode GPUs. The hardware underneath it is 8 GPU prefill machines and decode supernodes of 64 to 128 GPUs.

The trick that makes the number possible

The reason a fleet average reaches 14 million tokens per GPU-second is not that each request is served that fast. It is that most requests never reach the model in full.

The same document describes bounded replay: 2,560 tokens of context reconstructed as 128. The decode worker keeps a 3FS SSD cache holding KV state for three to seven days, so a returning conversation starts from remembered state instead of reprocessing its history. The cache does the heavy lifting, and a cache hit costs a fraction of a cold request.

There is a second detail that cuts against naive scaling. On speculative decoding, measured at 16, 60 and 96 concurrent requests, it loses 1.5 percent at 16, gains 5.4 percent at 60, and gains 11.0 percent at 96. The technique only pays at high concurrency. Below that threshold you are paying for draft tokens that do not help.

Put together, the 155 trillion figure is a statement about a system that is enormous, heavily cached, and saturated with returning traffic. It is not a statement you can copy into a smaller business.

The unit that decides everything

So I built the model that answers the question I actually had. If a token factory is not going to be DeepSeek scale, what does the smallest viable version look like, and what does it earn?

Take 64 decode GPUs. Rent them at $1.89 per GPU-hour, which is what a B200 class card costs on the spot market as I write. Sell tokens at $0.25 per million. Then move one variable, utilisation, and watch the answer change:

Breakeven sits at 37.5 percent utilisation and it does not move when you change the GPU count, because the cost and the revenue scale together. That one number is the whole business. Not the model, not the hardware, not the headline tokens per day.

This is also why the small cluster question has a blunt answer. Eight GPUs is not a smaller version of the business, it is a loss with extra steps. The efficient unit is 128 GPUs, and below that the utilisation needed to break even is not achievable with real traffic. Either commit to the unit, or rent someone else's.

What it costs to watch one thing, all day

The same arithmetic is worth running in the other direction, at personal scale. I have agents that check sources every day and report back. Forty checks a day at roughly 60,000 tokens each, with 70 percent of that served from cache, is about $5.55 a month on an off-peak DeepSeek class model. The same workload on a frontier model with cached input pricing is about $223 a month. Forty times the cost for a job that does not need frontier reasoning.

The lesson is the same one the utilisation curve teaches. Price per token is not the number that matters. What matters is how much of your work is cache hits, how much of the day your hardware is busy, and whether the task actually needs the expensive model.

Where you can check this yourself

I did not want to argue from a spreadsheet you could not inspect, so the model lives in public.

The AI factory calculator on my GPU tools site has sliders for GPU count, utilisation, throughput, GPU cost and selling price. The throughput presets are honest bands rather than one magic number: 5,000 tokens per second per GPU blended, 15,000 for decode heavy, 80,000 for prefill heavy, and the fleet averages are labelled as fleet averages.

Every input is a flag you can change, because my $1.89 and my throughput assumptions are inputs, not measurements. Set it to 64 GPUs, 20 percent utilisation and $0.25 per million tokens and it returns the minus 87.5 percent above. Move utilisation to 40 percent and it turns positive. That is the whole argument in one screen.

What I would tell someone starting

Three things, in order of how much money they save you.

Rent before you buy. A 128 GPU commitment is two and a half to three million dollars of hardware, or roughly $210,000 a month rented. The renting option tells you your real utilisation before you own the problem.

Measure your cache hit rate before you believe any cost projection. The 155 trillion number is a cache story. If your workload is cold and single turn, your economics are two orders of magnitude worse than the headline.

Treat utilisation as the product. Every point below 37.5 percent is a subsidy you are paying to your customers, and every point above it is margin. That is a sales and scheduling problem, not an infrastructure problem, and most token factory plans spend all their attention on the wrong one.