Watch our router pick the fastest provider, in real time.
The traffic split below is our latency router exploiting the fastest provider while auditing the rest, refreshed every minute from production traffic. Enable latency routing on your API key to get routed like this.
How the router decides
Measure every request
Each production request records time to first token and total latency per provider, model, region, stream mode, and input size. Stats are outlier-trimmed and recency-weighted with a 10 minute half-life, so a provider that degrades shows up within minutes.
Sample, don't average
For each request, the router draws one latency sample from every candidate's live distribution (Thompson Sampling) and routes to the lowest draw. Providers with little data get wider distributions, so new candidates keep getting explored without dedicated canary traffic.
Penalize unreliability
Candidates returning capacity errors (429s and 5xxs) over the last 15 minutes have their samples inflated by a penalty factor, up to 10x. An unreliable provider loses routing share immediately and wins it back as soon as it recovers.
Reading the board
Win share is the percentage of simulated routing decisions each candidate wins against the others in its group, using the same sampler the router runs in production. Latency is the geometric mean of total response time over the last hour. The winner is not always the lowest average: a candidate with volatile latency or recent capacity errors loses share to a steadier one. Groups split by streaming mode and input size because provider rankings flip between them.
/public/v1/routing-leaderboard