Codex vs Claude Code: How Much Does the Harness Matter?

How much does the harness change pass rate and cost in coding agents?

The harness changes cost far more than it changes pass rate. In the Frontier Harness Eval, one model (Kimi K3) ran the same 30 tasks through nine coding-agent harnesses. Codex led at 66.7%. Claude Code and DeepSeek Harness both landed at 63.3%. Yet DeepSeek Harness cost about $3.28 per successful task and Claude Code about $18.34. Across all configurations, the cheapest and most expensive differed by roughly 17.5x.

What is the Frontier Harness Eval?

It is a benchmark that isolates the harness. A harness is the software around the model: the agent loop, tool handling, context and caching behavior, permissions, and session management. Most benchmarks change the model and keep the scaffolding fixed. This one does the reverse.

According to the results as presented, the setup was:

The write-up that surfaced these numbers came from a creator walking through the published chart. The figures belong to the benchmark's authors, so check the original table for the full list of configurations.

What were the results?

The headline numbers are narrow on success and wide on cost.

The cheapest and most expensive configurations across the whole table sat about 17.5x apart. The DeepSeek Harness versus Claude Code comparison is smaller, around 5.6x, but still large for two runs with identical pass rates and an identical model.

Cost per successful task is the number to watch. It folds together token usage, caching, retries, and failed attempts into one figure. A harness that burns tokens on exploration or re-reads context inefficiently pays for it even when it eventually succeeds.

Why would the same model behave so differently?

The harness decides what the model sees and how often it is called. Several things move cost and success without touching model weights:

• Context handling. What gets loaded, trimmed, or re-sent on each turn drives input token volume.
• Caching. Harnesses that keep prompts stable can reuse cached tokens. Ones that reshuffle context cannot.
• Tool design. The tools on offer, and how their output is formatted, shape how many steps a task takes.
• Loop structure. Planning modes, sub-agents, and retry logic add calls. Sometimes they add accuracy. Sometimes they add only spend.
• Speed. Blocking behavior and call patterns change wall-clock time as well as dollars.

The source's summary was that the harness is not just a wrapper. It affects cost, caching, speed, and success rate even when the model is fixed.

Is the comparison fair?

Not entirely, and the benchmark's framing admits as much. Claude Code is optimized for Anthropic's models and Codex for OpenAI's. Running Kimi K3 inside either puts the model outside the setting its harness was tuned for. A harness that is tolerant of arbitrary models may look better than one built tightly around a single provider's behavior.

That does not make the result useless. Plenty of teams run open or third-party models inside general-purpose harnesses, and for them this is the realistic question. It does mean the table says less about how each harness performs with its native model.

Can you trust a 30-task benchmark?

Trust the shape, not the ranking. With 30 tasks, each task moves the pass rate by about 3.3 points. 66.7% is 20 of 30 and 63.3% is 19 of 30. Codex's lead over Claude Code and DeepSeek Harness is one task. Rerun the suite and the order could easily change.

The cost gap is a different matter. A 17.5x spread is far larger than the noise a single flipped task would introduce, so the conclusion that harnesses diverge sharply on cost is much sturdier than any claim about who wins on pass rate. Still, cost figures from 30 tasks depend on which tasks were chosen, and the source does not describe their difficulty mix.

What is DeepSeek Harness, and why did it score well on cost?

DeepSeek Harness (DSH) was released on August 13 alongside DeepSeek V4 Pro. It is MIT licensed, starts from a single npx command, and has a desktop app and web UI. It is not locked to DeepSeek models and works with other providers and OpenAI-compatible endpoints, which is why it can run Kimi K3 at all.

Its design principle is that everything is a plugin. The router, tool registry, session log, and even the agent loop are plugins rather than fixed core. It sits on a framework called Cordis, which aims for clean removal of components (unloading a plugin reverses what it registered) and dependency-aware activation.

Configuration is layered: bundles, then profiles, then a home-level patch, then command-line overrides. Each layer can target a plugin by ID and replace or add config. The UI exposes modes such as standard, minimal, and creator, which are just different plugin sets. In the eval, the creator configuration is the one that scored 63.3%.

The source does not say why DSH was cheap in this eval, and the benchmark data it cites does not break down token use. A modular design makes it easy to strip out expensive behavior, but that is an inference, not a measured result.

Is the harness more important than the model?

For many practical decisions, it is at least as important. The creator behind the source video said he put that question to two CTOs (Instacart and Box), and both leaned toward the harness. That is anecdote, not data. The eval gives a firmer data point: with the model fixed, outcomes still varied a lot on cost.

The caution runs both ways. A better model can lift every harness, and these results are for one model on one task set. The finding is that the harness is a first-order variable, not that models stopped mattering.

How should you evaluate a model or harness?

A few habits follow from the results:

1. Test a model in at least two or three harnesses. Otherwise you may blame the model for harness behavior.
2. Track cost per successful task, not only pass rate and not only cost per run.
3. Use enough tasks. Thirty is a start. Differences of one or two tasks are noise.
4. Run your own tasks. Repos, languages, and test setups differ, and harness strengths shift with them.
5. Record caching and retry behavior, since those often explain the cost gaps.