Same weights, twelve providers, 4.8x price gap: the provider variance problem

Open weights were supposed to commoditize inference. What they did instead was move the variance one layer down, from the model to the provider serving it.

A snapshot from this week makes the point better than any argument. On July 20 the same DeepSeek V4 Pro weights were listed across twelve providers at between $0.44 and $2.10 per million tokens, with uptime that varied just as much. That is a 4.8x spread on an identical model id.

It is not a DeepSeek quirk. On r/openrouter, someone noticed several providers pricing GLM-5.2 at $0.24 per million input tokens and asked the obvious question: promo, or new normal. Meanwhile Kimi K3 demand was so high after launch that one engineer reported running at 11 tokens per second on a popular gateway, and another said Kimi 3 was unusable through Vercel AI Gateway because of errors and latency.

Same weights. Wildly different product.

Why the spread exists

Providers serving open weights are not running the same stack. The variables that move price and behavior:

• Hardware generation. Older accelerators are cheaper per hour and slower per token.
• Quantization. FP8 or NVFP4 versus BF16 changes both cost and, sometimes, output quality on precision sensitive work.
• Batching policy. Aggressive batching cuts unit cost and raises time to first token under load.
• Context support. Some endpoints for the same model cap context far below the model's real limit.
• KV cache and prompt caching. Whether repeat prefixes are cached is worth more than the headline price on agent workloads.
• Capacity headroom. The cheapest endpoint is usually the most oversubscribed at peak, which is where the 429s live.
• Region. Data residency and network path both matter, and both cost money.

None of that is visible in a model string. You find it in production.

The metric almost everyone uses is wrong

The best Reddit thread of the week on this was blunt: tokens per second does not predict which model finishes the task first. The author cited provider variance on Kimi and GLM-5.2 and argued for selecting on seconds to finish, then pinning the provider.

That matches what we see in gateway data. Two failure patterns dominate:

Cheap but slow. A provider at half the price that takes three times as long. For a batch job, fine. For a coding agent with a human waiting, or for an agent loop where every step blocks the next, that trade is negative. A 40 step agent run at 11 tokens per second is not a cost saving, it is an abandoned session.

Cheap but flaky. A provider that serves 95 percent of requests well and fails the rest with timeouts or malformed tool calls. Your retry logic hides it, so it shows up as a mysterious 1.4x cost multiplier and a latency long tail. Someone testing Kimi K3 for a day wrote up exactly this: reasoning token cost, latency, and peak hour 429s.

The number to optimize is cost per completed task and wall clock per completed task. Everything else is a proxy that breaks under load.

How to route when providers vary this much

This is the part a gateway should do for you, because doing it by hand means maintaining a private latency and price table for hundreds of endpoints.

1. Route on latency, not just price. Latency routing picks the fastest healthy endpoint for a model at request time, which is the only way to track peak hour degradation without a human watching dashboards. The measurement approach is in how to route LLM requests by cost and latency.

2. Spread traffic and demote bad endpoints. Load balancing policies split traffic across acceptable providers for a model so one oversubscribed endpoint cannot become your ceiling. Pair it with fallback policies so a 429 or a 500 moves on instead of retrying into the same wall. We covered the statistics of reliability aware selection in reliability aware routing with Thompson sampling.

3. Pin when correctness depends on it. If a workload is sensitive to precision or tool call formatting, pin the provider rather than accepting whichever endpoint is cheapest today. For fixed capacity and predictable throughput, dedicated models remove the shared queue entirely.

4. Cache the repeated prefix. Agent traffic re sends the same system prompt and tool definitions on every step. Auto caching turns that from a per step cost into a one time cost, and it changes provider ranking, because a slightly pricier endpoint with caching often beats a cheaper one without it.

5. Measure per provider, not per model. Performance monitoring and usage analytics split by provider are what let you say "this endpoint got worse on Tuesday" instead of "the model feels off lately". Add request metadata so you can slice by workload too.

6. Watch tool calls specifically. Provider variance shows up first in structured output and function calling. Tool call analytics and structured outputs exist because the same model can be reliable at JSON on one host and shaky on another. Background reading in structured outputs across LLM providers.

What this means for the open weights argument

The bull case for open weights was that anyone can serve them, so price falls to the cost of compute. That is happening. Vercel published data showing open weight models at 29 percent of production tokens on under 4 percent of spend.

The part the bull case missed is that a commodity with twelve suppliers and no quality standard needs a broker. Someone has to know which supplier is fast today, which one is throttling, which one quietly caps context, and which one is worth a 20 percent premium because it caches. That is routing work, and it does not go away when the model is free. If anything the supply side explosion makes it heavier, which is the argument we made in the supply side explosion.

Practical next step: take your top three models by spend and check whether you are pinned to one provider by accident. Then compare the alternatives on the model catalog and set a policy instead of a default. If you want the routing side in one page, start with the quickstart.