What is an LLM router, and do you need one?

An LLM router chooses a model or provider for each AI request, using rules or a learned decision. As of November 2026, Requesty's coding-policy replay offers one reason to test routing: a Jev-based policy solved 120 of 129 tasks at 72% of the cost of sending every task to Claude Opus 5.5. Routing is worth testing when tasks vary in difficulty, model costs are high, or you need a fallback.

A router chooses where each request goes

Your application sends a request to the router, whose policy selects a model or provider. The selected model returns a response; if a configured attempt fails, a fallback policy can try another destination.

The policy can be a plain priority list, or it can use a classifier to choose a model for the task. AWS describes an approach called semantic routing: embed the prompt, compare it with reference prompts from task categories, then send it to the model associated with the closest category.

A router can sit inside a gateway, which is the broader service. Alongside routing, the gateway can provide a common API, authentication, budgets and usage tracking. For that broader API and access-control layer, see our guide to what an LLM gateway does.

Do you need an LLM router?

A single-provider prototype with acceptable cost and reliability can start with a direct integration. Adding a router is worth testing when you have a specific reason to send requests somewhere else.

Routing methods use different signals

These categories overlap: a decision model can itself be a learned model. The distinction helps when setting up the policy, since an explicit priority list, a preference-trained classifier and a typed decision call give you different signals to work with.

Take RouteLLM: it includes a matrix factorization router and a BERT classifier trained on preference data.

Requesty's Auto Routing beta, announced in October 2026, uses a decision model to classify the first user message against user-defined routes. If the decision misses the configured confidence threshold, it uses a fallback. The selected route stays pinned for the session.

What can routing change in cost and quality?

Model choice can change both the bill and the number of tasks solved. In Requesty's coding benchmark, 25 models worked on the same 129 private backend tasks with the same coding agent, tools and 80-turn budget. A task counted as solved only when its hidden tests and existing tests passed.

The gap depends on task difficulty. Haiku solved 100% of the easy and medium tasks and 55.6% of the extreme tasks; Opus solved 92.9% of the easy tasks, 100% of the medium tasks and 77.8% of the extreme tasks. The benchmark had 14 easy, 28 medium, 39 hard, 30 extra-hard and 18 extreme tasks.

Cost per task includes retries, so token price alone won't tell you the cost per solve. Cache behavior and token use also affect the bill: Haiku had a 100% cache-hit rate and used 1.66 million tokens per task in this benchmark; measure your own workload.

What did the Jev policy do?

In our coding-policy experiment, Jev 1.14 classified the kind of change using codebase context and four labels. The policy sent three of those labels to Haiku 5.5. Across all 129 tasks, that policy solved 120 at 72% of the cost of all Opus, which solved 119.

The decision calls were real production calls, but most of the comparison looked up previously measured agent outcomes instead of rerunning the agents for every policy. On the locked set of 43 tasks, the policy solved 40 and all Opus solved 41; we described those results as matching within noise.

Another decision model in the experiment, Cloudflare Clef, illustrates the tradeoff. Its policy solved 115 of 129 tasks at 57% of the all-Opus cost. The lower bill came with fewer solved tasks, so the right choice depends on what errors cost in your application.

Can routing improve reliability?

Fallback answers a different question: what happens when the chosen destination fails? Requesty fallback policies try configured models in priority order, with configured retries before moving to the next model.

Our eventual-success data counts a request as successful if an attempt in its fallback chain succeeds. The managed-fallback and direct cohorts had these results:

The data note compares observed production cohorts that chose different policies. That is a reason to measure eventual success across attempts in your own application, alongside error rates and latency. For more on provider failures, see our guide to rate limits and multi-provider fallback.

Latency needs its own measurement. Requesty's live latency rankings show latency per model and provider from production traffic, which is the signal our latency routing uses to pick a provider.

Start with a baseline

1. 1
   Measure one model

   Choose a set of tasks and record whether the model succeeds, along with cost per task and latency. Keep that baseline for comparison with each routing policy.

2. 2
   Select candidates

   Test other models on the same tasks. Check cost per solve as well as cost per task, and consider which models are acceptable fallbacks when a provider fails.

3. 3
   Choose a policy

   Start with explicit rules if you need a fallback or want to distribute traffic. For a decision-model policy, define the routes and test whether its classifications improve the result.

4. 4
   Configure fallback

   Set the model priority and retries in a fallback policy. Test what your application returns when the primary destination fails.

5. 5
   Measure the routed workload

   Compare solved tasks, cost, latency, cache behavior and errors with the baseline. User feedback can help you review quality after a routing decision.

Requesty's quickstart connects an application through an OpenAI-compatible API, and the model catalog covers 600+ models. You can test Auto Routing for session-level decisions, or use weighted load balancing and latency routing for operational policies.

If you have location requirements, check the model inference region separately from the EU gateway region. When estimating new usage, include the 5% pay-as-you-go markup on model costs.