Watch our router pick the fastest provider, in real time.
The traffic split below is our latency router exploiting the fastest provider while auditing the rest, refreshed every minute from production traffic. Enable latency routing on your API key to get routed like this.
How the router decides
Measure every request
Each production request records time to first token and total latency per provider, model, region, stream mode, and input size. Stats are outlier-trimmed and recency-weighted with a 10 minute half-life, so a provider that degrades shows up within minutes.
Sample, don't average
For each request, the router draws one latency sample from every candidate's live distribution (Thompson Sampling) and routes to the lowest draw. Providers with little data get wider distributions, so new candidates keep getting explored without dedicated canary traffic.
Penalize unreliability
Candidates returning capacity errors (429s and 5xxs) over the last 15 minutes have their samples inflated by a penalty factor, up to 10x. An unreliable provider loses routing share immediately and wins it back as soon as it recovers.
Reading the board
Win share is the percentage of simulated routing decisions each candidate wins against the others in its group, using the same sampler the router runs in production. Latency is the geometric mean of total response time over the last hour. The winner is not always the lowest average: a candidate with volatile latency or recent capacity errors loses share to a steadier one. Groups split by streaming mode and input size because provider rankings flip between them.
How "saved per request" is calculated
For each family with two or more providers, the achieved latency is every candidate's median weighted by the share of traffic the router sent it. The comparison is the plain average of the same medians, which is what a team that commits to one provider without knowing which is fastest should expect. The difference, weighted by each family's request volume over the last 24 hours, is the saving per request. It is not compared against the single fastest provider, since that one is only known in hindsight and changes through the day.
Price, families and what is left off
The $/mtok column is the blended price billed per million tokens on that provider over the rankings window, prompt cache discounts included, the same figure as the rankings table. It is not a one hour spot price and not a list price. Spellings of one model are folded into one family, image generation models are left off because their tokens and prices are not comparable with text models, and a candidate above 10% capacity errors is flagged as unreliable. Families served by one provider have no contest and are listed separately below the matchups.