The same benchmark suite costs $0.05 on one model and $3.69 on another: stop comparing price per token

Every price comparison you have read ranks models by dollars per million tokens. That number tells you the rate. It does not tell you the bill.

Artificial Analysis publishes something better: the measured cost of running their intelligence index suite on each model. Same work, same evaluations, measured spend. On the 2026-09-03 snapshot that figure runs from $0.049 on GPT-5.6 Luna to $3.689 on Claude Fable 5.1, a 76x spread.

The ranking is not the same as the per token ranking, and that difference is the whole point.

Where the two rankings disagree

Three inversions are worth staring at.

gpt-oss-120b has the cheapest input price on this list and is not the cheapest to run. At $0.15 per million it undercuts GPT-5.6 Luna's $0.20, yet the suite costs $0.073 against Luna's $0.049, and it scores 24.1 against Luna's 52.3. Cheap tokens, expensive work, weak result.

GLM 5.3 Flash reaches 57.5 for $0.087. Claude Fable 5.1 reaches 65.7 for $3.689. That is 42x the cost for 8.2 index points. Whether that trade is worth it depends entirely on how many of your requests genuinely need those points.

GPT-6 Astra and Claude Fable 5.1 publish the same $10.00 input price and differ by 2.2x on the bill, at $1.667 against $3.689. Identical rate card, very different spend, because they do not spend the same number of tokens getting to an answer.

Why the rate card lies

Two mechanisms drive the gap between price per token and cost per task.

Token appetite. Reasoning models emit thinking tokens you pay for and never read. A model that takes four times as many tokens to reach the same answer costs four times as much at the same rate. The rate card cannot show you this, because it is a property of the model's behaviour, not its price.

Cache economics. Long agent contexts get replayed, and cache read prices vary from a 98% discount down to no discount at all, which we cover in the cache read price comparison. Two models at the same headline input rate can differ several fold once most of your input is cached.

Neither shows up in dollars per million tokens. Both show up in your invoice.

What to do with this

Measure your own cost per task. Fix a representative set of jobs, run each candidate over the identical set, and compare total spend per completed job. This is the only comparison that reflects your prompts, your context sizes and your cache hit rate. Requesty analytics records per request spend so the numerator is already collected.

Put the expensive model where the points matter. The 42x gap between GLM 5.3 Flash and Claude Fable 5.1 is a strong argument for routing by cost rather than one global default: cheap capable models for the routine majority, the frontier model for the requests that genuinely need 65 rather than 57.

Cap the downside. A verbose model on a runaway loop is how a small rate becomes a large bill. Spend caps and alerts on spend spikes bound the failure instead of discovering it monthly.

Caveats worth stating

This is one benchmark suite, and its token mix is not your token mix. A model that looks efficient here can look wasteful on your prompts, particularly if your work is long context or heavily cached, since the suite has its own ratio of fresh to cached input.

The figures are a 2026-09-03 snapshot. Prices change, and several of these models changed price in the last month.

The comparison is also effort dependent. Artificial Analysis reports multiple effort settings per release and these are the strongest published setting, which is the most expensive one. Lower effort settings cost less and score less.

None of that undermines the core point. Price per token is a rate, not a cost. Rank your candidates by what the work costs, and the shortlist changes.