Three models can share a benchmark score and still require completely different supervision. Two tiny Go tasks showed why.
August 2026 · Juho KoskelaI have a growing problem with model leaderboards.
Not because benchmarks are useless. Quite the opposite: I spend an unhealthy amount of time building and looking at them.
Rather, the problem is what happens after a benchmark turns a model into a number.
As I write this, Artificial Analysis gives DeepSeek V4 Flash 0731 at max effort, GPT-5.6 Luna at max effort and Qwen3.8-27B at xhigh the exact same Intelligence Index score: 52.
What exactly does that mean?
It doesn’t mean those configurations used equivalent test-time compute; they don’t. And even before running these benchmarks, I wouldn’t have used those three models interchangeably for almost anything.
What I increasingly want to know about a model is not only how capable is it?
I want to know: what kind of supervision does it require when it fails?
To get a better feel for that, I built two small Go maintenance tasks. And I mean small: neither reference implementation was an impressive feat of software engineering.
Still, the models failed in remarkably different ways.
Kimi diagnosed the same bug correctly three times and then broke the public API three times. Sol independently discovered the right API three times, then generalized the fix further than I wanted. Qwen3.8-27B understood the underlying mechanics, never established the actual contract, and spent tons of tokens elaborating the mistake.
At some point, “which model is more intelligent?” stops being the useful question.
The first task involved Jira in a microservice that brokers tool calls from an internal chatbot to other company services.
jira.createIssue could create ordinary issues but couldn’t put an issue under another issue or an epic. The visible request deliberately did not specify exactly what the new API should look like.
The hidden contract was one optional parent parameter accepting either a Jira key like "AI-16" or a bare positive integer ID. null and omission meant no parent. Invalid values should fail normally.
The repository already contained a parser implementing almost exactly the same string-or-number convention elsewhere.
The implementation itself was trivial – the actual task was noticing that you didn’t yet know what to implement.
The second task came from my open-source Pipedrive Go SDK.
Files.Add accepted an arbitrary caller-built multipart body. That meant an upload from a non-seekable reader couldn’t be replayed after a 429, because the HTTP request had no way to reconstruct its body.
The repository already contained another upload API accepting a filename and io.Reader. The frozen intended fix was roughly thirty production lines:
Upload(ctx, fileName, content, ...) convenience method.Add.Add alone.Jira tested uncertainty. Pipedrive tested restraint.
I chose them partly because those are different failure modes. The fact that rankings move between two deliberately different tasks demonstrates that task sensitivity can be large; it does not tell us how often some arbitrary distribution of coding tasks would reorder a leaderboard.
The more interesting part is what individual models repeatedly did.
Before this accidentally acquires the authority of a scientific paper: this is two deliberately chosen tasks, with three runs per model configuration.
It is a case study in how coding models can fail differently, not an estimate of general coding intelligence.
The headline comparisons use the same xhigh reasoning setting. That doesn’t mean each model got a similar amount of test-time compute – that’s something I can’t control.
I also ran a few models at their maximum reasoning settings, mostly because I was curious what additional test-time compute would do.
Not much, at least consistently, as it turns out.
Each contestant received only the visible task and an isolated repository checkout.
I also built a small user simulator for one specific reason: I wanted the agent to be able to stop coding and ask about genuinely ambiguous product decisions.
When an agent ended its turn with a clarification question, the simulator saw that final message and could answer it before the agent continued. It did not see the contestant’s patch or tool trajectory.
The simulator knew a fact sheet containing the hidden task contract. If the agent asked a question covered by that fact sheet, it answered it. If the question was something the agent should discover from the repository, such as where a helper lived, the answer was quite blunt:
Use your judgment.
That means the mechanism has an obvious limitation: a useful question buried somewhere in tool work but omitted from the agent’s final message is invisible to the simulator.
Still, it let me test something most coding benchmarks don’t: does the agent recognize when it should stop implementing and ask the user?
The simulator was GPT-5.4-mini. I also tested more expensive models including Sonnet 5 and noted no practical drift from 5.4-mini’s behavior.
Scoring happened in three layers:
Deterministic gates checked compilation, tests, compatibility and the hidden contract.
I hand-graded each patch against a frozen specification, using Fable 5 and GPT-5.6 Sol as additional judging and review aids. I knew which model produced each run, so this is explicitly a maintainer evaluation, not a blinded academic benchmark.
The weighted rubric was:
| Dimension | Weight |
|---|---|
| Implementation correctness and contract drift | 30 % |
| Idiomatic Go | 25 % |
| Execution and trajectory | 20 % |
| Tests, documentation and scope | 15 % |
| Tool efficiency | 10 % |
“Taste” is a dangerous word because it sounds like I’m deducting points for brace placement. I’m not.
In this case it mostly means: does this patch look like something a competent maintainer would actually want?
There is an unavoidable gap in what I can publish; The Jira task comes from a private repository. I can’t release the repository, patches or raw trajectories. I can publish the aggregate results and methodology here:
Pipedrive is different. The repository is public, and the golden answer has since reached GitHub anyway, so the task is already contaminated. For that task I’m publishing the full trail:
Full Pipedrive leaderboard · Task specification and scoring materials · Raw agent runs · Pairwise judgments
Consider the Pipedrive task retired.
Across the full Jira sweep, including the additional max-effort runs, I had 51 graded attempts.
Two passed the hidden contract. Zero reused the existing parser.
Nearly every model was capable of finding Jira’s fields.parent and writing perfectly respectable Go around it.
That wasn’t where they failed; they failed by deciding what the API should be without establishing what the user wanted.
I got:
parent_keyparent_issue_keyparent_key and parent_idAll because the visible request did not specify one parameter shape.
“Uncertainty kept turning into implementation.”
The especially revealing failures were the ones that looked good.
Several models correctly understood Jira IDs versus keys. Several wrote useful tests. Some updated documentation. A few inspected adjacent code containing the exact convention they needed.
Then they tested the service directly, carefully proving their invented API worked, while completely missing the JSON request boundary the hidden task actually exercised.
Fable produced the strongest individual Jira result after the simulator corrected its initial parent_key assumption. Opus 4.6 also produced a very strong corrected result.
That’s the key finding here: being able to implement a clarified contract is not the same capability as recognizing that the contract needs clarification.
The Pipedrive task had almost the opposite problem.
Most models correctly diagnosed the bug: a non-replayable body means no safe retry. Buffer the generated multipart body and the resulting request can be reconstructed.
Great! Let’s implement the fix.
Then everybody started having ideas. Some changed the existing Add API. Some added a generic request-body replay framework.
Some changed low-level transport semantics. Some introduced new option hierarchies and file associations.
Some duplicated all of Add’s HTTP request and response handling inside the new method.
The frozen reference was still sitting there waiting patiently:
contentType, body, err := multipartbody.NewFile("file", fileName, content)
if err != nil {
return nil, err
}
return s.Add(ctx, body, contentType, opts...)Kimi K3 is my favorite example – it diagnosed the replayability problem correctly in all three runs.
It also broke the stable public API in all three runs.
That is an impressively clean demonstration that debugging ability and API-design judgment are not the same capability.
Sol did much better. It independently inferred the required new Upload API in all three runs, reused the multipart machinery and passed every deterministic gate.
It also decided, in all three runs, to make Add itself buffer every request body.
That’s very clever. It fixes retryability for everything. It also silently changes a stable public streaming API so arbitrary bodies now get pulled into memory.
Good solution, wrong patch. I guess some sympathy is called for, one of its design choices did land very close to the where the production API finally ended up.
Suffering from success, I suppose. But this is exactly why scope matters:
being right about what an API should eventually become does not make that functionality right for today’s patch.
Fable had the same broad instinct in another form.
It understood the design space extremely well, but repeatedly widened transport behavior and added hundreds of lines around a problem that wanted a narrow SDK change.
Which lines up rather well with my broader experience of the model: excellent design instincts, mediocre implementation restraint.
When Fable shines, it really shines. When it doesn’t, your thirty-line patch has acquired an architecture.
Several behaviors were strikingly consistent even across three runs:
| Model | Repeated behavior in these runs |
|---|---|
| GPT-5.6 Sol | Found the requested Upload API independently 3/3 times; also generalized retryability into Add 3/3 times |
| Kimi K3 | Diagnosed the replayability problem 3/3 times; changed the stable API 3/3 times |
| GPT-5.6 Terra | Repeatedly committed early to the wrong public contract on both tasks |
| GPT-5.6 Luna | Repeatedly found the simple implementation pattern but failed to investigate the contract deeply enough |
| Qwen3.8-27B | Missed the required public contract in every run on both tasks while producing unusually large trajectories |
At least inside this tiny test chamber, some failure modes repeated. That is more interesting to me than whether a model ranked fourth or fifth.
It also rhymes with my experience using these models for coding tasks daily. That’s where my more subjective takes come from, not from pretending six benchmark samples constitute a model’s personality.
Fable tends to search for the elegant architectural answer and then keep going. Sometimes, though, Fable really does one-shot the task.
Sol tends to understand difficult problems quickly and sometimes solves a slightly more general problem than I asked for – or writes tons of unnecessary code.
Kimi is unusually good at finding ways something can break, which is one reason I increasingly like it as an adversarial reviewer.
Qwen3.8 Max has plenty of capability and remarkably little fear of scope creep.
And Qwen3.8-27B repeatedly gives me the strangest combination: it can understand the local mechanics while failing to frame the actual task correctly.
For transparency, I served Qwen3.8-27B unquantized at full BF16 accuracy on a 180 GB B200. Quantization is not a concern.
“Models aren’t only differently capable. They’re differently wrong.”
The Pipedrive task exposed another problem – my own rubric still compressed away too much information.
So I took every candidate pair and ran a separate blinded merge-quality evaluation: 276 pairs, three judgments each, 828 votes in total.
The question was essentially which of these patches would you rather merge?
Big caveat: the judge was GPT-5.6 Sol at xhigh effort. I see you raising your pitchfork.
Model identity was hidden. But that does not make the result neutral ground truth.
Blinding prevents explicit model favoritism; it does not remove family or stylistic preference. And yes: Sol also judged patches produced by Sol.
I therefore treat this as a Sol-judged view of merge quality, not the final word on software aesthetics.
The results were still very useful. All three Sol runs passed the mandatory gates.
They also averaged a Bradley-Terry-implied 0.2% probability of beating the reference patch.
Why?
Because the tests could establish that buffering Add fixed retryability.
They could not express strongly enough that I did not want an existing streaming API to acquire that behavior.
DeepSeek V4 Pro produced almost the mirror image; only one of its three runs passed the mandatory gates.
But that one passing patch was the strongest contestant patch in the pairwise evaluation, at 1302 Elo against a reference fixed at 1500, corresponding to a 24.3 % Bradley-Terry-implied probability of beating it.
That was rather surprising to me – DeepSeek showed a very high ceiling and terrible reliability on this task.
Opus 4.6 was different again. Its corrected patches were strong in the merge-quality evaluation even though the model repeatedly needed help establishing the contract in the first place.
So now I had three evaluations answering different questions:
| Layer | Question |
|---|---|
| Deterministic gates | Does the patch satisfy the required behavior? |
| Maintainer rubric | How good was the implementation and engineering process? |
| Pairwise review | Which patch would I actually rather merge? |
They all disagreed.
Which was rather inconvenient, because I built this benchmark partly because I thought existing scalar scores threw away too much information.
Then my own scalar score threw away too much information.
Fair enough.
Full pairwise results and judgments
This brings me back to the three models I started with.
Artificial Analysis currently assigns an Intelligence Index score of 52 to DeepSeek V4 Flash 0731 at max effort, GPT-5.6 Luna at max effort and Qwen3.8-27B at xhigh. Source
That score may be useful for exactly what it claims to summarize. In these tasks, the models did not resemble interchangeable engineering tools.
DeepSeek Flash was fast and mechanically capable, but repeatedly showed weak API judgment and scope control.
Luna tended toward cleaner and simpler implementations, but did not investigate ambiguous contracts deeply enough to solve these tasks reliably.
Qwen3.8-27B understood much of the local mechanics, then repeatedly spent enormous trajectories constructing sophisticated answers to the wrong problem.
Same number – very different thing to supervise.
“A benchmark can measure its target well and still be a poor model-selection tool.”
A scalar necessarily throws information away and sometimes that’s exactly what you want. If I’m comparing twenty models at a glance, I would rather have one number than a twelve-dimensional tensor.
The problem starts when the compressed number gets mistaken for the model.
The practical result for me is believe it or not, not build a new leaderboard.
It’s more task-specific routing. My broader use of these models already pushed me in that direction, these two tasks just gave me clean examples of why.
For difficult architecture and API-design questions, I usually have Fable involved. That’s where its urge to search for the elegant solution is useful.
Then I freeze the decision before Fable finds three other abstractions to redesign.
For implementation, Sol is usually a very strong fit. It executes well, moves efficiently, and its tendency to generalize largely goes away when a contract is explicit.
For adversarial review, I like Kimi. I don’t necessarily want it redesigning the API. I very much want it trying to find everything wrong with the implementation.
And for bigger architectural changes, Fable can complain about the result again. So one workflow I increasingly like looks roughly like:
Fable designs → freeze the contract → Sol builds → Kimi tries to break it → Fable complains about the architecture.
It seems the future of software engineering may involve assembling a team of artificial coworkers with intentionally incompatible personality defects.
The question I care about is shifting from:
Which model is best?
toward something more like:
Which model do I want making this particular kind of mistake, and how much supervision will it take to catch it?
Price complicates the routing again, because token price, token consumption, wall-clock time and useful work are four very different numbers.
The detailed leaderboards include those too.
Full Jira leaderboard · Full Pipedrive leaderboard
That’s another article. For this one, the conclusion is simpler.
A leaderboard has to put these models in an order.
I’m increasingly not convinced I should.
A benchmark score is not a model.