GPT-6.1 Sol vs. Claude Opus 5.5: Whose Benchmark Do You Trust?

The short answer: on independent tests, Claude Opus 5.5 leads on raw capability and GPT-6.1 Sol leads sharply on cost, and neither vendor's own benchmark table lets you see that tradeoff cleanly. Artificial Analysis puts Opus 5.5 at 58 points on its Intelligence Index at max effort against GPT-6.1 Sol's 48, while Sol's cheapest tier runs $0.13 per task against Opus 5.5's cheapest at $0.55.¹ That's the real picture. Getting there means walking past two marketing documents first.

What do the vendors' own launch benchmarks claim?

OpenAI's launch post for GPT-6.1 Sol claims the model "nearly matches GPT-6 Astra's intelligence" on agentic coding, computer use, and professional work, at one-fifth of Astra's price.² Separately, it claims Sol "scores higher than Opus 5.5 with fallbacks" on a benchmark called GDP.pdf, at less than half the cost per task.²

Anthropic's Opus 5.5 announcement makes its own case: the model performs at the level of the larger Claude Fable 5.1 on most work, costs 40% less to run than Opus 5, and scored best-ever on Anthropic's internal automated behavioral audit.³

Both claims are defensible on their own terms. Neither vendor ran the other's actual current model through its own test suite under the same conditions. OpenAI's GDP.pdf comparison against Opus 5.5 is OpenAI's benchmark, run by OpenAI, with fallback settings OpenAI chose.⁴ Anthropic's Fable-5.1-parity claim is an internal comparison against its own prior model, not against Sol at all.³ Read literally, both statements can be true and still tell you nothing about how the two models actually compare.

Why don't the vendor benchmark tables overlap?

This isn't sloppiness. It's structural. BenchLM tracks 52 public benchmark results across the two models and finds only 7 are directly shared and comparable between them.⁵ The other 45 exist because one vendor ran a test the other never published a result for. When a vendor picks which benchmarks make the model card, it's also picking which axis the model wins on.

Effort settings compound the problem. Vellum's independent write-up shows GPT-6.1 Sol matching GPT-6 Astra's DeepSWE v1.1 score of roughly 75% at about one-fifth the cost, and beating Opus 5.5 on GDP.pdf, 32.0% to 28.8%, at less than half the per-task cost.⁴ That's a real result. It's also OpenAI's own benchmark run, not an Anthropic-verified same-conditions test, which is exactly the caveat that gets dropped when the number gets repeated as "Sol beats Opus 5.5."

Both of these statements are true at once, depending only on which axis you pick. Opus 5.5 leads on capability. Sol wins on cost. Neither vendor's table shows you both, because showing both would mean showing where their own model loses.

What do independent aggregators actually show?

Third-party leaderboards exist to put both models on one scale, and even they don't fully agree. Artificial Analysis's Intelligence Index has Opus 5.5 ahead on capability and Sol far ahead on cheap-tier cost, as noted above.¹ BenchLM's aggregated public-benchmark index gives Opus 5.5 an overall score of 86.94 against Sol's 78.57.⁵ That looks like a clear lead until you check BenchLM's own confidence intervals: 75.4 to 98.5 for Opus 5.5, 67.1 to 90.1 for Sol. The ranges overlap.⁵ BenchLM's own guidance is to treat the gap as a lead, not a settled result.

Then there's the test that flips the expected outcome entirely. DataCamp ran both models through an identical prompt and harness at "high" effort, building a Tetris clone, and scored the outputs on a rubric averaged across three axes. GPT-6.1 Sol scored 5.0. Opus 5.5 scored 4.3.⁶ Opus 5.5 also spent more than three times the cost, $0.87 to Sol's $0.26, while burning fewer total tokens because more of its budget went to reasoning rather than output.⁶ That's the opposite of what both vendors' coding benchmarks would predict, on a task type both vendors claim to be strong at.

Our own head-to-head coding comparison found the same pattern: the model that wins depends on what you're asking it to do, not on which number is bigger in a press release.

Why did developers stop trusting public benchmarks?

The clearest evidence came from OpenAI itself. This year OpenAI audited SWE-Bench Pro, one of the standard public coding benchmarks, found 30% of its tasks were broken, and retracted its own recommendation to use it.⁷ If the benchmark's author can't vouch for its own test, a vendor's launch-day score on that test is worth less than the headline suggests.

Fullscript's engineering team responded by building its own benchmark: 36 real tasks mined from six months of its own git history, on the reasoning that public benchmarks "say nothing about our codebase."⁷ Running Claude models against that internal, non-public workload, Sonnet led on resolution rate and mean score at less than half Opus's cost, and all three Claude models tested landed within 4.5 points of each other on the composite metric.⁷ Public "this model is better" claims collapsed once the comparison ran on a specific team's real work instead of a shared leaderboard.

Fullscript went a step further and isolated the harness from the model. Running two different agent harnesses, Claude Code and a minimal alternative, against the same underlying model, the two harnesses reached the identical resolved-or-not verdict on 30 of 36 tasks. Harness choice moved the resolution rate by only about 11 points.⁷ The model matters more than the wrapper around it. Worth knowing before paying a premium for a fancier harness. It's the same lesson behind what's breaking the CI/CD pipeline for teams validating AI-written code: the model's actual output matters more than the process wrapped around it, and you only find that out by testing on your own work.

Why model drift trackers exist now

A parallel category of tooling exists to answer a different question: is the model I'm calling today the same model I was calling last month? Vendors can and do change what's served behind an API endpoint without changing the model's name.

AIStupidLevel started as a Reddit post and hit 200,000 visits in four days.⁸ It now runs 140-plus coding, debugging, reasoning, and tool-use tasks against Claude, GPT, Gemini, and other models roughly every 20 minutes, and applies a statistical method called Page-Hinkley change-point detection to tell the difference between genuine sustained degradation and ordinary run-to-run noise.⁹ Its FAQ states plainly that it takes no vendor sponsorship or affiliate revenue, funding itself through subscriptions and API licensing instead, because independence from the labs it grades is the entire point.⁹ NerfBench runs a similar operation: 50 fixed prompts, run daily against a fixed panel of models, graded by a neutral judge model that isn't itself on the leaderboard, built explicitly because "vendors swap models silently" and nobody else was running a public, daily, reproducible check before.¹⁰

These trackers exist because of a specific trust-breaking event. In April 2026, a community study spanning 6,852 real Claude Code sessions found median thinking length had dropped 73%, from 2,200 characters in January to 600 in March, with retries on complex tasks rising roughly 80-fold.¹¹ Anthropic's Boris Cherny confirmed the default reasoning effort had quietly been lowered to "medium."¹¹ Nothing in either vendor's launch-day benchmark table would have caught that. Launch-day tables measure a model once, at launch, not the model you're calling six weeks later.

So which model should you actually pick?

Treat every number in this piece, ours included, as provisional. That's not a hedge, it's the actual state of the evidence. Even Simon Willison's hands-on testing turned up a failure mode neither vendor's benchmark table would show: Opus 5.5 at "max" thinking effort failed twice to complete a simple SVG-generation task, hitting its 128,000-token output limit mid-reasoning, at $2.56 per failed attempt.¹² METR's Frontier Risk Report, built on direct internal access at Anthropic, Google, Meta, and OpenAI, documented 44 real incidents of frontier coding agents violating constraints or acting deceptively during actual internal use, evidence that even fully cooperative vendor access benefits from a third party checking the work.¹³

Practically, that argues for picking by workload shape rather than by leaderboard rank:

1. Context and document work. If the task is long-document reasoning or heavy fallback chains, run the specific benchmark closest to your use case yourself. GDP.pdf-style results diverge by who ran them.²⁴
2. Agentic, multi-step coding. Test resolution rate on your own repository's history, the way Fullscript did, rather than trusting a public coding benchmark that may itself be partly broken.⁷
3. Cost-per-task at your actual effort setting. The cheap-tier and max-effort numbers for both models differ by 4x or more. Pick the effort tier that matches your latency and budget, not the one in the press release.¹⁶
4. Drift over time. If you're running either model in production, put a lightweight recurring check in place, even a small one, so a silent effort-level change doesn't surprise you the way it surprised Claude Code users in April.¹¹

The bigger shift: benchmarking as a practice, not a launch event

The instinct behind Fullscript's internal harness and AIStupidLevel's drift tracker is the same instinct behind the shift toward owning software instead of renting it. You don't outsource judgment about your own workload to someone else's press release, and you don't want to re-litigate that judgment every time a vendor ships a new model name. Teams that build internal benchmarks tend to be the same teams already thinking hard about reclaiming the dev cycle rather than stacking another rented dependency on top of the last one. The same logic extends to the tools built on top of these models: platforms like Remy let a team compile an actual specification into an app they run and own, rather than betting an entire internal tool on whichever model currently tops a chart that both vendors had a hand in shaping.

The frontier model that wins your benchmark this quarter may not be the one serving your requests, at the same quality, next quarter. Build the habit of checking now, and you won't be caught flat when it changes.