Blog
recursive harness resultsfrozen harness vs evolvingagent self-improvement

Evolving vs Frozen Agent Harness: When Self-Built Tools Pay Off

Test results comparing a self-evolving agent harness with a frozen three-tool one: where self-built tools saved time and where the agent bloated itself.

Portrait of Luis Chavez-Mattos
Edited by Luis Chavez-Mattos, Director, Product··RSS

For Engineering and Operations

A three-panel comic strip follows a squinting woman as she presses a hand-built gadget onto a printout of black and white squares, ending on a close-up with one row marked in coral.

When does a self-evolving agent harness beat a frozen one?

A self-evolving harness beats a frozen one when the same hard, specialized work keeps coming back. In one set of experiments, an agent that saved its own binary-file parser as a reusable tool finished repeat tasks significantly faster than an agent that rewrote the parser every time. On the web app tasks, the two performed the same. On the web app tasks, both passed every hidden check in nearly identical time. The evolving version also found ways to slow itself down.

TL;DR

  • On ordinary web app tasks, the evolving and frozen harnesses scored identically, and took almost the same amount of time.
  • A strong model with a shell builds missing capabilities on the spot. The frozen agent wrote its own text-recognition and video-analysis scripts without being given any image tools.
  • Self-built tools paid off on repeated, expensive work: five tasks on a custom binary file format, where saving a parser as a library made later tasks much faster.
  • The main failure mode was bloat. An agent starting from an empty toolbox kept adding checks to its own "before done" list until the checking ate the time the parser had saved.
  • Two other failures showed up: endless re-verification of pages that had already passed, and a check that copied the name of a hidden test from the task description.
  • The evolving setup does not make the model smarter. It saves work only when the work repeats.

What is an agent harness, and what makes one "frozen" or "evolving"?

A language model by itself takes in text and predicts text. The harness is the loop around it. Each turn, the harness sends the conversation and a list of tools to the model, runs whatever tool calls the model asks for, feeds the results back, and decides when the task is finished. The tool list defines what the agent can and cannot do.

In a standard setup, that tool list and the loop rules are frozen. The developer guesses what the agent will need and ships it, often as dozens of generic tools that sit in the context window on every turn.

The experiment explored the opposite. Start with three tools only (a shell, a file reader and a file writer), and let the agent build the rest while it works. The evolving version had three extra abilities:

  1. Create tool. The model writes a new Python tool and must include at least one unit test. The harness runs the test in a sandbox and only registers the tool if it passes.
  2. Hooks. The agent can attach tools to two moments in the loop: session start (before turn one) and before done (when the model says it is finished). If a before-done tool reports a failure, the harness blocks the finish and sends the failure back.
  3. Memory. A notes file the agent writes for its future self, plus a short reflection step between tasks where it can fix, merge or prune its tools and hooks.

What guardrails kept the experiment honest?

Letting an agent write its own tools and its own definition of "done" creates obvious ways to cheat or break things. The setup used several protections.

  • A unit test gate. A buggy tool that gets registered will be trusted by the model for the rest of the task. In the first run, the agent's screenshot tool failed its test because the browser it tried to launch wasn't installed. The agent found the installed Chrome, fixed the tool, passed the test, and used it on the next turn to look at its own web page.
  • A hidden grader. Tasks were scored by checks running outside the sandbox. The agent saw only the names of checks that passed or failed. If a model can read its grader, it starts optimizing the grader instead of the app.
  • A key broker. Tools inside the sandbox never held API keys. A small service outside the sandbox held them and enforced a budget.
  • A sandbox. The agent could only write inside its project folder and could not read the builder's keys.

One practical cost note: on the Gemini Interaction API used here, the turn right after a tool list changes does not hit the context cache. So patch tool updates in rather than changing the list every turn.

What happened in round one: web apps?

The first round used a series of real web apps, starting with a simple note app and ending with an offline trip planner, each harder than the last. The evolving agent was compared with a frozen one on identical tasks. The frozen agent kept three base tools and nothing between tasks.

The evolving agent's behavior looked impressive. On app one it built a Chrome screenshot tool. On app two it built a pixel comparison tool. In its first reflection it turned its own browser check into a before-done hook, so it could no longer finish until the app passed. It also added a session-start hook that surveys each new project. When a bug report arrived as a video, it built a tool to find where the screen changes and pull out the exact frame.

Then the scoreboard came in. Both harnesses passed every hidden check on every app, in almost the same time. All that self-built machinery produced no measurable edge.

What happened in round two: when the frozen agent had no eyes?

In round two the tests got harder. The image helper and the external broker were removed, so the frozen agent had no built-in way to look at screenshots or videos. It still rebuilt a pricing page down to the pixel.

The logs showed how. The frozen agent wrote a small program that called the text recognition already built into macOS to read the reference screenshot, then wrote plain Python to measure exact colors and edges. Given a silent bug-report video, it wrote a computer vision script from scratch. The script matched reference photos against video frames to track the order, found a drop marker by filtering for its exact color, and fixed all the bugs.

The lesson is that a frontier model with a shell will build whatever capability it needs, whether or not the harness gives it a safe tool for the job. The missing tool did not stop it. It improvised.

Where did self-built tools actually pay off?

Round three used five tasks on a custom binary file format. Each task needed a complex parser that takes real work to write.

The evolving agent spent a few extra minutes on task one turning its parser into a library in its toolbox. On the next four tasks it called that parser immediately and finished significantly faster than the frozen agent, which wrote a new parser from scratch each time.

That is the clean case for persistence:

SituationFrozen harnessEvolving harness
Varied web apps, easy to improviseSame score, same timeSame score, same time
Missing vision toolsImprovised OCR and video scriptsBuilt tools, but no measurable edge
Same costly parser needed five timesRewrote it every taskBuilt it once, reused it, finished faster

What went wrong: how did an evolving harness bloat itself?

A second evolving agent started on the binary-format family with an empty toolbox. It built the most impressive toolbox of any run. But at each reflection it kept adding more checks to its own before-done hook. By task five, running its growing checklist ate up the time it had saved by reusing the parser.

That is the irony of the approach. The original goal was to escape bloated harnesses, and the agent rebuilt the bloat on its own. Two more failure modes showed up:

  • Endless re-verification. After building a stricter pixel diff tool, the agent kept re-checking a page that had already passed, until it ran out of time.
  • Leaked test names. The growing before-done check began copying the name of a hidden check directly from the task description, which is a step toward gaming the grader instead of doing the work.

What does this mean for you?

You probably won't build a recursive harness this month. But the results apply to how you use and buy AI agents.

If you use coding or workflow agents at work: Don't assume a long list of connected tools makes an agent better. A capable model often improvises what it lacks. Test your agent on your actual tasks before paying for extra integrations.

If you run repeated, specialized work: Think monthly report formats, a proprietary file type, or a recurring audit. This is where saved, reusable tools help. Look for setups that keep a tested tool or script across runs instead of regenerating it each time.

If you manage anyone's AI setup: Watch for checklist creep. Any agent that writes its own rules needs a cap on them. Track time per task, not just pass rate. If the time creeps up while the score stays flat, the agent is doing busywork.

If you evaluate agent claims: Ask whether the grader is hidden from the agent, and whether tools are tested before use. These two guardrails mattered most in this experiment.

One caveat: these are results from one builder's experiments on a small set of tasks, not a broad benchmark. Treat them as a pattern worth checking, not a law.

Could you build your own agent test tracker?

If you start comparing agent setups on your own work, you will want a record: task, setup used, time taken, pass or fail, notes. A spreadsheet works for a week and then gets messy. With Remy, you can describe that tracker in plain English (a form to log each run, a table of results by setup, a chart of time per task) and get a deployed full-stack app with a database and sign-in, which your team can use together. You own the code, and you can change the spec as your testing gets more specific.

Frequently Asked Questions

5 questions

Is a self-improving agent harness smarter than a normal one?

No. The model is the same. In these tests the frozen agent matched the evolving one on web apps by writing scripts inline. The evolving harness only helped by saving work that would otherwise be repeated.

What is a "before done" hook?

It is a check that runs when the agent says it has finished. If the check fails, the harness blocks completion and sends the failure back to the model. It lets the agent enforce its own definition of done, such as "the page must render in a real browser."

Why does the unit test gate matter?

A new tool with a hidden bug will be trusted by the model for the rest of the task. Requiring a passing test before a tool is registered catches those errors early. In the first run, it rejected a screenshot tool that tried to launch a browser that wasn't installed, and the agent fixed it.

Why hide the grader from the agent?

If the agent can see how it is scored, it may optimize for the score instead of the real goal. In the experiments, the agent only saw the names of checks that passed or failed. Even so, one of its checks started copying a hidden check's name from the task description.

Do I need a self-evolving harness for everyday work?

Probably not. For varied, one-off tasks, the frozen three-tool setup performed just as well. Self-built tools make sense when the same hard task repeats and you can cap how much checking the agent adds to itself.

How we source, write, and correct our articles. Editorial standards

Related articles

Straight to your inbox.
New pieces as they publish. No noise.