Record on Terminal-Bench, catastrophic on a private eval: Fable 5.1 and the case for your own benchmark

Anthropic's 1 September release notes for Fable 5.1 are specific and, as far as anyone can tell, accurate. 52.6% on Terminal-Bench-Science 0.1, more than double Fable 5. 55.8% on Terminal-Bench 4.0 against 42.0%. A new standard across their benchmark suite.

A day later a developer who maintains a private stress test against every Claude release posted this:

TL;DR: Fable 5.1 failed my personal benchmark in a way no other Claude model has.

Their full account is worth reading because the methodology is better than most published evals. One long dictated prompt with about a dozen embedded traps: contradictory math, garbled words to resolve, a referenced attachment that was never sent, deliberately impossible formatting instructions, all under custom preferences requiring tool calls for logging and file verification. Every frontier model for months had aced it. Fable 5, Opus 5 and Sonnet 5 all scored at or near perfect at every effort level.

Fable 5.1 produced their worst recorded result. And the failure shape is the interesting part: both runs did all the tool work, and the thinking summaries showed it had understood the prompt correctly, catching the math error and resolving the garbled dictation. Then it delivered essentially nothing.

The model did the reasoning and dropped the output.

Both results are true

There is no contradiction to resolve. A benchmark measures a distribution of tasks. Terminal-Bench-Science and this person's trap suite measure different things, and a release can move sharply in opposite directions on each.

That is not a scandal, it is the expected behaviour of models tuned on some objectives and not others. What makes it consequential is that most teams choose models from the first number and discover the second number in production.

The same three days produced a second, independent version of the same lesson. A developer who moved from a frontier model to an open weight alternative wrote that the experience showed how much people gaslight themselves based on a number or symbol, naming version strings and launch graphics as the thing doing the persuading. And a commenter on the GPT-6 launch made the same point arithmetically: remove the name and the launch hype and the independent numbers look a lot less revolutionary.

Three people, three different models, one conclusion: the published number is not the thing you are buying.

Why the hype cycle is so predictable

Someone called the arc on launch day, before it happened:

Days 1 to 3: "THIS IS AMAZING, it one-shot my entire refactor." Days 4 to 7: "THIS IS THE WORST MODEL EVER SHIPPED, Anthropic has ruined it."

That pattern is not fickleness. It is what happens when a population evaluates a model by vibes on a self selected sample. Early adopters try easy wins and are delighted, a genuine reaction: one user described pasting in an inherited 2020 codebase and getting a correct diagnosis of a timezone bug buried four functions deep that three people had missed. Then the same population hits the awkward cases, and with no fixed task set nobody can tell a real regression from a harder sample.

An eval suite is what converts that noise into a decision. Anthropic's subreddit needed a release discussion hub to contain the volume of launch reaction, and almost none of that reaction is measurement.

Build the smallest useful eval set

You do not need a research programme. Twenty to fifty tasks, and the discipline to run them.

Take tasks from your own traffic. Sample real requests, including the ones that broke. Every production incident is a permanent test case.

Score outcomes, not prose. Did the tool call fire with the right arguments, did the file change, did the JSON validate, did the number come out right. This is where the Fable 5.1 failure would have been caught instantly: tool calls fired, reasoning was correct, final output was empty. A rubric grading only the answer text would have scored it zero and a rubric grading only tool calls would have scored it perfect.

Include the traps deliberately. Contradictory instructions, a referenced attachment that does not exist, impossible formatting, deliberately messy input. Frontier models handle clean prompts. Differences show up on malformed ones, which is what your users send.

Vary effort explicitly. The private test above ran Low and Medium and failed at both, which is a stronger result than a single setting. Effort interacts with quality in ways that are not monotonic, and a cheaper setting frequently wins. Use reasoning controls to sweep it.

Pin the versions you compare. Model identity is not stable: vendors route named models to updated builds without announcement. Log the resolved model for every request through request metadata and logs so an eval result is attributable to a specific build rather than to a name.

Make replay cheap, or you will not do it

The reason most teams skip this is friction. If pointing your suite at a new model means a code change, it will not happen on launch day, which is precisely when you need it.

The fix is to make model choice configuration. Put a managed policy between your harness and the model so a candidate is one edit, run the suite, and read cost and latency and outcome from usage analytics and session reconstruction rather than from impressions. Keep approved models as the gate so nothing reaches production until it has passed, and fallback policies so a bad promotion degrades instead of breaking.

Then a launch becomes a twenty minute exercise: point, replay, compare, decide. We argued the same thing about release cadence in the model release treadmill, and the last week made the case better than we did.

The wider pattern

Our own gateway data has shown for months that no model stays number one and that the supply side keeps growing faster than anyone can evaluate it. Add the September releases and the arithmetic is unforgiving: nobody can hand test every candidate.

Which leaves two options. Choose models from vendor benchmarks and marketing, and absorb the variance in production. Or own a small, boring, outcome scored task set and let it make the decision in twenty minutes.

The developer who caught a catastrophic regression the day after a record breaking launch was not smarter than the labs. They just had a fixed ruler.

For our own measured comparisons see the best AI coding model on benchmarks and cost and agentic routing benchmarked, and browse current models and prices in the models catalog.

Start routing on Requesty and make replaying your eval a config change.