Why Did Open Models Fall 12 Points Behind in Index v5?
Artificial Analysis Intelligence Index v5 uses 60% private tests and harder agentic coding. The open-model gap grew from 3 to 12 points. Here is what it means.
Why did open models fall 12 points behind the frontier?
Open models did not get worse. The test changed. Version 5 of the Artificial Analysis Intelligence Index now runs 60% of its evaluations on private test sets or held-out answers, and it adds much harder agentic tasks. Under the new scoring, the best downloadable open models land at 32, below Claude Haiku at 36. The gap to the frontier went from about 3 points to 12.
TL;DR
- The gap widened from 3 to 12 points because the benchmark was rebuilt, not because open models regressed.
- 60% of the index now uses private test sets or held-out answers, which limits the effect of questions leaking into training data.
- The new coding test, Frontier Code 2, only credits a change a repo maintainer would merge, and one failed blocking check scores zero.
- On the new Science Agent benchmark, the best downloadable open model scores about 13%, most score under 10%, and top closed models sit around 60.
- The tasks are far more agentic: the median model now writes 59,000 tokens per task, up from 35,000.
- If you pick a local model from an old leaderboard, treat those rankings as possibly flattering and test on your own workload.
What changed in Artificial Analysis Intelligence Index v5?
Three changes matter most.
| Change | What it does |
|---|---|
| 60% private or held-out tests | Questions and answers are not public, so they cannot leak into training sets |
| Frontier Code 2 | Scores coding work by whether a repo maintainer would merge it |
| Science Agent benchmark | Tests agentic scientific work rather than static question answering |
The index is a composite. When you swap out a large share of its components for harder ones, every model's number moves. The frontier closed models absorbed that shift better than the open ones did, which is why the spread grew.
How does Frontier Code 2 score coding?
Most coding benchmarks ask whether code passes a set of tests. Frontier Code 2 asks a stricter question: would a maintainer of the repository merge this change?
That framing is closer to real engineering. A patch can run and still be unmergeable because it breaks a convention, misses an edge case, or fails a review gate. Under this test, failing one blocking check gives you zero for that task. There is no partial credit for a mostly correct diff.
Open models fall off here. Binary, all-or-nothing scoring punishes small errors, and long multi-step coding work gives small errors many chances to appear.
How do open models score on the Science Agent benchmark?
Badly, relative to closed models. The best open model you can download scores only 13% on Science Agent. Most open models score under 10%. The top closed models are around 60.
That is a much bigger spread than the 12-point overall gap. It suggests the overall number understates how far apart the two groups are on long, tool-using, multi-step work. The aggregate blends in easier categories where open models still compete.
Does this mean open models were cheating on benchmarks?
No. The result does not show that anyone gamed anything. It shows that the old tests may have been easier to do well on.
Here is the mechanism. Public benchmarks publish their questions. Over time, those questions and sometimes their answers end up in web data that gets scraped into training corpora. A model can then look stronger than it is without any intent to cheat. This is benchmark contamination, and it is hard to rule out for any public test.
Private test sets cut that risk. If 60% of the index is held out, a model cannot have seen most of it.
The other factor is task design. The new tests are much more agentic. The median model now writes 59,000 tokens per task, up from 35,000. Longer tasks mean more steps, more tool calls, and more places to go wrong. Models that were tuned for short, well-defined questions pay for that.
Should I trust old leaderboards when picking a local model?
Use them as a rough filter, not a verdict. If a leaderboard relied heavily on public static tests, it may have flattered open models, and the v5 numbers hint at that. The 3-point gap was probably a mix of real capability and an easy test.
A few practical points:
- Old scores are still useful for comparing open models to each other on short, simple tasks.
- They are less reliable for judging open models against closed ones.
- They say little about agentic work such as long coding sessions or multi-step research.
- A model that ranks well on a leaderboard may still fail your specific workflow.
How should I evaluate a local model now?
Build a small private test from your own work. Keep it out of any public repo so it cannot leak.
- Collect 10 to 20 real tasks you actually run, such as bug fixes, refactors, or document summaries.
- Define a pass rule up front. For code, borrow the maintainer standard: would you merge it as is?
- Run each candidate model on the same tasks with the same prompts and tools.
- Record failures, not just average quality. One blocking error should count as a fail.
- Re-run when you change quantization, context length, or the agent harness, since those can shift results.
Compare cost and speed too. An agentic task that burns 59,000 tokens takes real time on local hardware, so throughput matters as much as the score.
Is a 32 vs 36 gap a reason to avoid open models?
Not by itself. A 32 against Claude Haiku's 36 is a modest difference on a composite index. Many local use cases, like private data handling, offline work, fixed costs, and simple automation, do not need frontier agentic performance.
The weak spot is long-horizon agentic work, where Science Agent shows a wide spread. If your workload looks like that, test carefully before committing. If it is short-form drafting, classification, or extraction, the headline gap may barely matter.
Frequently Asked Questions
What is the Artificial Analysis Intelligence Index?
It is a composite benchmark that combines scores from multiple evaluations into one number so models can be compared. Version 5 changed its mix of tests, which shifted scores across the board.
Did open models actually get worse?
No. The models did not change. The index did. Harder, more agentic, and more private tests exposed a larger gap than the earlier version showed.
What is Frontier Code 2?
It is the new coding test in the index. It counts a change only if a repo maintainer would merge it, and failing any blocking check scores zero for that task.
Which open model scored 32?
The sources describe the result as the best downloadable open models scoring 32 overall, without naming a specific model here. Check the Artificial Analysis site for the current model-level breakdown.
How we source, write, and correct our articles. Editorial standards
