Why one benchmark won't tell you the best coding LLM in 2026 — and which three together actually do

You want to refactor a 50,000-line codebase. You go look up which LLM is best at coding right now. LMSYS Arena tells you GPT-4o is on top. HumanEval has been saturated for two years and tells you everyone is on top. Aider Polyglot ranks GPT-5 first, but the latest 2026 frontier models are missing entirely. SWE-bench Verified says Claude Opus 4.7 is the leader. A vendor blog post quotes their own model winning whatever benchmark they picked. They can't all be right, and the truth is none of them are wrong — they're measuring different things, and you've been treating them like they measure the same thing.

This is the single most expensive mistake in 2026 model selection. You pay for it in failed refactors, hallucinated patches that pass tests on the wrong file, and quiet hours spent re-prompting a model that simply isn't the right tool for the job. The fix isn't to find one perfect benchmark. The fix is to learn to read three together.

"Coding" is not one task

Before you can pick a leaderboard, you have to admit that "coding" is at least three different tasks with three different difficulty curves.

TASK                          WHAT IT REWARDS                EXAMPLE BENCH
─────────────────────────────────────────────────────────────────────────
Single-file edit              clean diffs, format adherence  Aider Polyglot
Repo-wide patch (agentic)     navigation, multi-file logic   SWE-bench
General reasoning + code      breadth, math, science, code   AA Intelligence
Snippet from prompt           pure code generation           HumanEval (solved)

Single-file edit is what you do in Cursor or Aider all day: "find this function, change this argument, don't touch anything else." It rewards a model that produces minimal, surgically correct diffs in a strict format. A model that generates a beautiful 100-line rewrite when the right answer is a 3-line change fails this task even though the code "works."

Repo-wide patch is what an autonomous PR agent does: given a GitHub issue and a whole repo, navigate to the relevant files, understand the cross-file logic, write a patch that passes the project's existing tests. This rewards a completely different skill — context juggling, repo memory, restraint about what not to touch.

General reasoning + code mixed is what happens when you ask a model "explain why this query is slow, then write the fix." It needs to reason about indexes, understand the query planner, and also produce correct SQL. This task lives in the same place as math and science benchmarks, because the failure mode is misunderstanding the problem, not mis-typing the code.

Snippet from prompt — the classic HumanEval task — has been effectively solved since 2024. Every frontier model and most decent OSS models score above 90%. If a leaderboard is ranking models by HumanEval in 2026, ignore that leaderboard.

A model that's #1 at one of these can be middle-of-the-pack at another. This isn't a leaderboard flaw. It's a fact about the underlying capability.

The three benchmarks I actually trust

After dropping the saturated and the misleading, three sources survive. Each one is best-in-class for its task and useless for the others.

Aider Polyglot — for editing

Aider runs 225 Exercism problems across six languages (C++, Go, Java, JavaScript, Python, Rust) and requires the model to produce edit-format diffs that compile and pass tests. The brilliance of the benchmark is in its format strictness: a model that solves the problem in prose but won't output a clean unified diff fails. This is the closest public benchmark to your real Cursor/Aider workflow.

What it measures well: minimal, format-correct diffs. Multi-language coverage. What it doesn't measure: large codebase navigation, anything beyond a single file. Watch out for: the leaderboard updates slowly — by 2026-05, the newest 2026 frontier models often aren't on it yet because Paul Gauthier hasn't run them through. That's not a flaw; it's a feature of running an honest evaluation with real test execution.

SWE-bench Verified (via Vellum aggregator) — for repo work

SWE-bench gives the model a real GitHub repo + a real issue, and asks for a patch that resolves the bug without breaking the project's existing tests. Verified is the curated subset where humans confirmed the issue is solvable. This is the hardest agentic coding benchmark we have right now, and the only one that meaningfully captures "can this model be left alone to fix a bug in a codebase it didn't write."

Vellum's leaderboard aggregates vendor-announced SWE-bench Verified scores, which means it stays current with frontier releases far faster than Aider does. As of today, the SWE-bench rows it lists are almost entirely Claude/GPT family — that's the reality of which labs are bothering to evaluate on this benchmark, not a Vellum bias.

What it measures well: autonomous coding agent capability. Reading large code. Restraint. What it doesn't measure: language coverage (it's Python-heavy), edit format, anything sub-repo. Watch out for: short list. If your candidate model isn't listed, it usually means no public SWE-bench score exists yet.

Artificial Analysis Intelligence Index — for breadth

This is the fastest-moving public LLM ranking on the internet right now. AA aggregates MMLU-Pro, GPQA Diamond, AIME 2025, HumanEval-style coding, and several other evals into a single composite. Within days of a major model release — GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro Preview, Grok 4.3, DeepSeek V4 Pro — AA has a number for it. That's a property no other public leaderboard has.

What it measures well: cross-domain reasoning, current 2026 model coverage, frontier vs OSS comparison. What it doesn't measure: any single task in depth. A high Intelligence Index doesn't guarantee strong repo-wide patching. Watch out for: composite scores hide the failure modes. A model can score well by being great at math and acceptable at coding rather than being great at coding.

Reading the three together — the composite heuristic

Here's the matrix you actually want. Sample row from today, showing where the three sources land for a single model:

Three observations:

1. The "winner" depends on the column you look at. GPT-5.5 wins AA. Claude Opus 4.7 wins SWE-bench. Gemini 2.5/3.1 Pro (which Aider tested as the gen-1 variant) wins Aider Polyglot. There is no model that wins all three.

2. Empty cells are not zeros — they're "not tested yet." Aider hasn't run GPT-5.5 or Claude Opus 4.7 through its harness. Vellum hasn't aggregated a 2026 SWE-bench score for Gemini or Grok. Treat these as missing data, not as evidence the model is weak there.

3. The cells that are filled give you a directional signal anyway. If Claude Opus 4.7 beats Sonnet 4.6 on SWE-bench (87.6 vs 82.0), and Sonnet leads in some other dimension, you can reasonably guess how the Opus row would fill in.

So how do you actually pick?

Notice what's not in the heuristic: vendor blog posts, X threads, and "I tried it and it felt better." Those have their place, but they're not how you make a defensible model choice.

Honest limitations

Three things this approach doesn't fix.

Benchmark lag is real. Aider's harness takes work to run, so the newest frontier models are routinely missing for 2-3 months after release. Vellum is faster but depends on labs publishing their SWE-bench numbers. AA is fastest but generic. If a 2026 model isn't on Aider yet, the Aider column will be empty — that's not a leaderboard failure, it's how honest evaluation works.

OSS models are under-covered on coding benches. Vellum's SWE-bench list is heavily Claude/GPT. Aider has stronger OSS coverage but lags. If you're choosing between Qwen3-Coder-480B and DeepSeek-V4-Pro for self-hosting, the public leaderboards will give you partial answers at best — at some point you have to spin them up and benchmark on your own task.

HTML scraping is fragile. AA and Vellum don't expose public APIs, so any pipeline that pulls daily scores from them depends on their HTML structure not changing. A redesign on either site breaks the pipeline until someone updates the parser. It's stable enough for now, but it's not a guarantee.

The page that does this for you

I built /leaderboard because doing this matrix by hand every time a new model dropped was getting expensive. It pulls Aider Polyglot from the GitHub raw YAML, SWE-bench rows from Vellum's JSON-LD, and Intelligence Index values from Artificial Analysis's table, normalizes them per column, and shows the same matrix above as a live page. The cron runs daily at 06:00 KST. Every score is verbatim from its source — no LLM-generated commentary on the numbers themselves, only on the methodology.

If you're picking a model right now: open the page, read the column that matches your task first, then glance at the other two for sanity. If you're tracking what's moving in the frontier: the page changes every day, AA leads the indicator.

If a fourth benchmark belongs on it, I'm listening. Two I'm watching: LiveBench (currently hard to parse cleanly) and a coding-specific subset of LMSYS Arena (currently noisy with chat preference). Both might earn a column if their data access improves.

The bigger lesson, though, isn't about which leaderboards. It's about the habit. Stop asking "which model is best at coding." Ask "best at which kind of coding." The answers get a lot more useful, and the gap between people who know which model to reach for and people who guess gets a lot wider.

2026.05.20