Routing coding agents with one question: what a decision model on the first message is worth

Coding agents spend most of their money on a few hard sessions, and most of their sessions on easy ones. A cheap model can finish the easy ones. The open problem is telling them apart before the agent starts, using only what the developer typed.

We tested one way to do that on Requesty today: a decision policy. The router sends the first user message of the session to a decision model with one multiple choice question, maps the answer to a model, and pins that model for the rest of the session. The decision model never sees a model name. It only answers a question about the work.

This post reports what that bought us on 129 coding tasks from our own Go monorepo, every variant we tried, the ones that failed, and the limits of the result.

Summary

1. One question matched Opus at 72% of the cost. Asking TypeSafe Jev 1.14 "what kind of change is this?" with four labels and one paragraph of codebase context, and sending three of the labels to Claude Haiku 5.5, solved 120 of 129 tasks. Claude Opus 5.5 on every task solved 119.
2. The signal is whether one change must land consistently in several places. Additive changes, renames and single place behaviour changes were safe on Haiku. Behaviour changes spread across handlers, directions or streaming paths were not.
3. The vague first message is enough. A one line request routed as well as the full four thousand character spec (120 versus 119 tasks solved).
4. More text did not help. A 624 character instruction with codebase context was the best variant. Longer instructions, label guides, AGENTS.md and a package map of the monorepo never solved more tasks.
5. Optimising the prompt against outcomes overfits fast. A self improving prompt loop that saw per task results scored +5 tasks against Opus on its training tasks and 32 of 43 on locked tasks, against 40 for the short question.
6. Jev kept Opus accuracy, three other decision models did not. With the same question and map, Cloudflare Clef, the Perplexity decider and Azure decision-1 saved more but solved 112 to 115 tasks.
7. Live runs agree with the replay. A live production run of a related first message gate on 90 tasks solved 80 against 81 for Opus, at 87% of the cost.
8. It does not transfer for free. On SWE-bench Verified and DeepSWE no generic question beat the strongest single model in the pool.

Setup

Tasks and grader

The benchmark is 129 tasks taken from real changes to the Requesty backend: Go services for the router, billing and dashboard API, plus shared libraries for provider adapters, schemas, pricing and routing. Each task has a hidden test suite. A task counts as solved when the agent's patch passes it. Tasks are split into five tiers by a human (easy 14, medium 28, hard 39, xhard 30, xtreme 18).

Every model in the pool had already been run once on every task with the same agent harness and grader. That gives a matrix of solved or not and cost per task for Claude Haiku 5.5, Claude Sonnet 5.5, Claude Opus 5.5 and GPT 6.1 Sol. Running a single model on every task gives:

An oracle that knew in advance which cheap model could solve each task would sit far to the left of Opus. A router can only approach that point if it can predict failures from the request.

Replay

Most results below are offline replays. For each routing policy we send each task's first message to the decision model, read the label, look up the model the policy maps that label to, and take that model's measured result and cost from the matrix. The decision calls are real production calls. The agent runs are not repeated, which lets us compare hundreds of policies on identical agent outcomes.

What the decision model sees

Real developers do not open a session with an issue and acceptance criteria. They type one or two sentences. For each task we kept a vague rewrite of the spec, for example "Hey, could you implement the Anthropic 1 hour prompt cache pricing in the Go backend". Unless stated otherwise, the decision model only sees that vague first message.

Guarding against overfitting

Choosing a map from labels to models on the same tasks you score it on flatters every policy. We used three guards:

• Fixed maps. For the final policies the map was written before scoring: easy labels to Haiku, the hard label to Opus.
• A locked split. 43 tasks were set aside at the start of the prompt search. No prompt proposer and no map ever saw their outcomes. Opus solves 41 of them.
• Random splits. For the question comparison, the map was fitted on a random half of the tasks and scored on the other half, 300 times.

How a decision policy works

The policy is one object. The instruction and the label keys are what the decision model reads. The values are what the router does with the answer.

{
  "models": [{ "model": "typesafe/jev-1.14.0" }],
  "instruction": "What kind of change is this? If the request lists several existing places that must all change, prefer the last label.",
  "branches": {
    "additive: new field, constant, helper, adapter or endpoint following an existing pattern": "anthropic/claude-haiku-5-5",
    "rename, move or refactor with no behaviour change": "anthropic/claude-haiku-5-5",
    "changes existing behaviour in one place": "anthropic/claude-haiku-5-5",
    "changes existing behaviour in several places (handlers, directions, streaming paths) that must all match": "anthropic/claude-opus-5-5"
  },
  "default_model": "anthropic/claude-opus-5-5",
  "min_threshold": 0.5
}

On the first request of a session the router:

1. Sends the first user message, the instruction and the labels to the decision model.
2. Reads the probability of each label and takes the most likely one.
3. Routes to that label's model if its probability is at least min_threshold, otherwise to default_model.
4. Pins the chosen model to the session, so every later turn of the same agent conversation goes to the same model without another decision call.

A failed or slow decision call also falls back to default_model. Setting the default to the strongest model means an outage of the decision model costs money, not tasks.

Experiment 1: which question to ask

We wrote ten questions about the first message and two controls, and asked each of them for all 129 tasks on both the vague and the full prompt: 2,580 Jev calls. The questions covered the domain of the change, the effort, the scope in files and packages, the risk, the clarity of the spec, a five level difficulty scale and the tier a reviewer would assign.

Two results stand out. Every question looks good in sample: between 119 and 122 tasks solved once the map is fitted on all 129 tasks. Held out, most of them lose two to seven tasks per 129 against Opus, and random labels lose 7.3. Questions about subject area (five or nine domains) are the most misleading: they look the cheapest in sample and lose four to five tasks out of sample.

The two questions that held were the scope of the change and the tier a reviewer would give it. Both are about how much of the codebase the change touches. That led to the question we kept, which asks what kind of change the request is and singles out one kind: a behaviour change that has to stay consistent in several existing places.

Experiment 2: the change kind question

The four labels are:

• additive: new field, constant, helper, adapter or endpoint following an existing pattern
• rename, move or refactor with no behaviour change
• changes existing behaviour in one place
• changes existing behaviour in several places (handlers, directions, streaming paths) that must all match

The map is fixed: the first three to Haiku, the last to Opus.

About half of the sessions go to Haiku. On those, Haiku is as good as Opus: in the "one place" bucket Haiku solved 34 of 35 tasks and Opus 33, and in the refactor bucket Haiku solved 5 of 5. The hard bucket is where models separate. Inside it Sonnet solves 51 of 63 and Opus 57, which is why sending it to Sonnet halves the cost again but gives up about five tasks per 129.

On the locked 43, the policy solved 40 against 41 for Opus. On the full 129 it solved one more than Opus. We read both as "matches Opus within noise", not as a win. A fresh rerun of Opus on the 48 hardest tasks moved its own score by three to five tasks.

Vague message or full spec

The short message routes as well as the full spec. Long specs list many files and identifiers, and Jev reads that as "several places" more often, which sends more sessions to Opus. For a product this is the useful direction: real first messages are short.

One practical note for agents such as Claude Code: their first user message also carries system reminders and environment context. In our tests sending those to the decision model made routing slightly worse and the call six times more expensive. The decision should read what the developer typed.

Experiment 3: does more context help?

A decision model like Jev accepts tens of thousands of tokens, and the instruction above is 118 characters. The obvious question is whether telling it much more about the codebase lets it judge better. We kept the four labels and the map fixed and changed only the instruction.

The best variant adds one paragraph: who is asking, what the monorepo contains, and that the agent will also have to write tests.

You are triaging the first message a developer sends to an autonomous coding agent working in the Requesty Go backend monorepo (golang/cmd: router request proxy, accounting billing, api-v2 dashboard API; golang/lib: provider adapters per LLM vendor, request/response schemas, pricing calculators, routing policies, guardrails, ClickHouse/Postgres data access). Read what the developer is actually asking for, infer what the agent will have to touch to deliver it including tests, and classify the request. What kind of change is this? If the request lists several existing places that must all change, prefer the last label.

It was three points cheaper than the bare question at the same accuracy. That difference is within noise, but it was never worse on any split.

Anything longer hurt. A map of all 395 Go packages made Jev call more requests "several places" (74 instead of 63), so cost rose with no extra tasks solved. The instruction is the same on every call; only the developer's message changes. Context helps only if it lets the model tell two requests apart, and a map of the whole repository does not say which part of it this request needs.

We also tried reframing the question around intent ("what is the developer asking for?") with the same context, with five labels, and with a step by step rubric. None of them beat the change kind question. The short reframing was cheaper (62% to 66% of Opus) but solved 118. The five label version solved 120 but cost 91% of Opus.

Experiment 4: a self improving prompt loop

We ran two automated prompt searches. A proposer model wrote candidate questions and maps, each candidate was scored on the training tasks, and the scores were fed back for the next round.

• Loop 1 saw only aggregate scores. Over 12 rounds and 80 candidates it converged on the change kind question.
• Loop 2 was asked to write long, detailed rubrics and was shown real training tasks together with which models solved them, to help it split the hard bucket for cheaper models.

Loop 2's best candidate, a 2,500 character rubric with five labels, scored five tasks above Opus on its training tasks at 24% of the cost. On the 43 locked tasks it solved 32. The short question from loop 1 solved 40.

Shown per task outcomes, the proposer wrote rules that described those tasks rather than the kind of work. The longer the allowed prompt, the more room it had to do that. With 129 tasks there is not enough data for a long, outcome tuned rubric to pay off. The locked split is the only reason we caught it.

Experiment 5: the confidence threshold

min_threshold sets how sure the decision model must be before its label is used. Below it the request goes to default_model.

Jev is rarely below 0.5 on this question, so 0.5 costs one point and only catches requests where it is unsure, such as "support the new model". From 0.6 up, every step sends confident easy sessions to Opus for no gain. We use 0.5.

Experiment 6: four decision models, same question

We sent the 129 vague first messages to four decision models through the production router, 516 calls in total, with the same question, labels and map.

Any two models agreed on the route (cheap or Opus) for 74% to 87% of tasks, and all four agreed on 83 of 129. Clef, the Perplexity decider and decision-1 agree with each other more than with Jev. Jev is the most cautious: it calls "several places" most often, which is exactly the label that protects accuracy.

This comparison is not neutral. The question was developed on Jev's answers. Each other model may need its own wording or map, and on a lower cost target they are already more aggressive. For a policy that must not lose tasks against Opus, Jev was the only one that held.

Experiment 7: live production runs

Replay assumes the agent behaves the same way when the model is chosen by a router. To check, we ran live arms in production through the same harness and hidden grader. The live arm closest to the policy above used the first vague message only, asked Jev for a difficulty score, and sent the session to Haiku below the cutoff and to Opus above it.

Of 36 sessions routed to Haiku, 35 were solved. All decision calls for the 90 sessions together cost $0.004. The saving comes from the easy and medium tiers, where Jev sends almost everything to Haiku and Haiku costs 6% to 37% of Opus. On the two hardest tiers it routed one task each, so those tiers cost about what Opus costs. The one task below Opus on xtreme is within the three to five task rerun noise we measured for Opus itself on the same 48 tasks.

The change kind policy itself was exercised end to end on the router in staging: requests reached Jev, hard asks were routed to Opus, and follow up turns of the same session stayed pinned to the chosen model.

Experiment 8: public benchmarks

We repeated the question comparison on two public sets with published per task results for several models: SWE-bench Verified (500 Python tasks) and DeepSWE (113 tasks in Go, Python, TypeScript, JavaScript and Rust). The decision model read each task's problem statement; maps were fitted on half and scored on the other half.

The result is negative. On SWE-bench Verified with a Claude Haiku, Sonnet and Opus pool, the control of running Opus on every task was the best policy: the best question, a long scope rubric, averaged 0.26 tasks below Opus at the same cost, and the rest lost more. On DeepSWE, with an OpenAI reasoning effort ladder, no question beat the strongest setting. Only on one Claude ladder did a question beat the control (0.64 tasks below the top setting at 81% of its cost, against 1.1 below at 82% for the control).

Our reading: on public sets the hard part of each task is specific to an unfamiliar repository and is not visible from the issue text. On our own monorepo, a question written in its vocabulary ("handlers, directions, streaming paths") separates the work a cheap model can finish. The gain comes from writing the question for the codebase, not from the decision model alone.

Limitations

• Sample size. 129 tasks, 43 locked. One task is less than one point of accuracy, and Opus reruns move by three to five tasks on the 48 hardest. Differences of one or two tasks between policies are not significant.
• One codebase. All tasks come from one Go monorepo. The question and the context paragraph are written for it.
• Replay versus live. The change kind numbers are replays over measured agent runs. The live arm used a related difficulty gate on the same first messages, not the exact same question.
• First message only. The decision is made once, before any tool call. Sessions that turn out harder than they looked stay on the cheap model. Asking the decision model again mid session did not help in our tests: it predicted eventual failure no better than chance and escalated almost every run.
• Cheaper hard models cost tasks. Sending the hard label to Sonnet or GPT 6.1 Sol costs 42% or 22% of Opus and gives up five to eight tasks per 129.

What we recommend

For a coding agent on a codebase you know:

1. Create a decision policy with TypeSafe Jev, the change kind labels above, and a one paragraph description of your codebase in the instruction.
2. Map the additive, refactor and single place labels to a cheap model, the "several places" label to your strongest model, and set default_model to the strongest model.
3. Set min_threshold to 0.5.
4. Send only what the developer typed to the decision call.
5. Hold out part of your own task history before you tune anything, and judge every change on it.

On our benchmark that kept Opus accuracy at 72% of its cost, for a routing call of a few hundredths of a cent per session. Getting further, for example by moving the hard bucket to a mid tier model without losing tasks, needs either more labelled sessions than 129 or a signal from inside the session, such as the first files the agent opens.

To start, see the Decisions guide, the Jev listing, and our earlier decision model benchmark.