How Can an AI Agent Improve Its Own Harness in Production?

What is harness engineering for production agents?

Harness engineering means improving everything around the model: the rules, skills, workflows, and guardrails. For a production agent, it also means tracking the mistakes the agent makes with real users, then going back to decide what to change in those rules and skills so the same mistake doesn't repeat. Coding-agent users already do this with files like CLAUDE.md or AGENTS.md. The same habit works for customer-facing and business-automation agents.

How do coding-agent harness ideas map to production agents?

The mapping is close enough to treat as a checklist. If you already manage a coding agent, you know the pieces.

The old approach put most of this in code: a Python function per tool, hand-wired guardrails, a prompt stitched in by hand. The newer approach keeps the harness in markdown files, which makes it easy to edit when behavior is wrong. If the agent misbehaves, you change a document instead of refactoring a pipeline.

What does a managed deep agent look like?

LangChain's managed deep agents are one platform that packages this setup. The creator who demonstrated it showed an agent.py of about 30 lines that defines the agent, including guardrails, sub agents, tools, skills, and instructions. Most of the substance lives in files that the platform picks up by convention:

• An instructions.md at the root of the agent folder, working like a CLAUDE.md or AGENTS.md.
• A skills folder, working like a coding agent's skills directory.
• A memory folder, which holds learned cases and a learned agents.md.

When you run locally or deploy, the platform stitches these pieces together. LangSmith provides tracing and metrics underneath.

You don't need this specific platform. You can build your own deployment and harness pipeline. The strategies below apply either way. What matters is that something bundles your markdown into a running agent and records what that agent does.

How does the escalation memory loop work?

The first loop gives the agent a way to learn lessons while it runs. It has four parts.

1. Detect uncertainty. When the agent is struggling with a user's question, it says so instead of bluffing. Models hallucinate, but they are surprisingly decent at surfacing their own uncertainty.
2. Escalate to a human inbox. The agent still tries to help the user. It does not block and wait. But the case lands in a team inbox with the question and the agent's answer.
3. Confirm or correct. A human either marks the answer correct or supplies the right one.
4. Write the lesson down. The agent stores the lesson in a context hub, which it can search later.

Retrieval matters here. The lessons are not dumped into the context window the way the main instructions are. The agent searches for relevant cases when it needs them. That keeps the prompt small while the knowledge base grows.

In the demo's memory folder, escalated cases are stored as files. There is also a core agents.md made of rules learned from those escalations, plus private memory per user. The creator noted that per-user memories never overlap and that users can't view each other's memories, which is enforced through the agent's guardrails.

What did the escalation demo show?

The demo used a help desk agent for a community, built on real questions the creator had received. The test had two steps.

First, a user named Nina asked about a local fix that wasn't being picked up when she ran a workflow. The agent searched her memories, loaded a diagnostic skill, and updated her memory. The whole run took about 10 seconds after local compilation. It answered that it didn't have enough to say for sure, and the team inbox count rose from four to five.

The human then supplied the fix: the tool creates its work trees from the remote main branch, so the change needs to be pushed before rerunning. About 10 seconds after confirmation, the agent reported a new shared memory and updated both the core agents.md and Nina's private memory.

Second, a different user, Omar, asked a related but not identical question: a run finished green and opened a pull request, but the diff was missing three local commits. This time the agent did not express uncertainty. It said it was likely not a bug and explained the underlying behavior, applying the earlier lesson to a similar case. That generalization is the point. The agent did not need an identical question to benefit.

How does the trace-driven improvement loop work?

Memory lessons are useful, but they are per-case notes. They are not the higher-level harness: the core workflows and instructions. The second loop improves those.

LangSmith Engine collects trace data from production runs and surfaces failures. In the demo, a failure entry included:

• A detailed description of what went wrong.
• Evidence, with a link to the exact LangSmith trace, so you can read the conversation and see what the model was doing when the issue happened.
• A proposed fix at the bottom of the entry.

The source material cuts off before showing what the proposed fix looked like in full, so treat the details of its format as something to check in the product itself. The creator described these as suggestions that can be applied automatically. If your risk tolerance is low, route them through review first.

How do you keep a human approving every change?

Both loops have a natural checkpoint, and you can make both strict.

• Loop one: the human supplies or confirms the answer before it becomes a lesson. Nothing enters shared memory without that step.
• Loop two: treat each Engine suggestion as a proposed change to a markdown file. Read the evidence trace, decide whether the diagnosis is right, and only then accept the edit. Because the harness is markdown, the change is a readable diff you can review like any code change.

Version control helps here. If the harness lives in files, you can review, revert, and compare changes the same way you do for a coding agent's rules.

Is this worth building for a production agent?

It makes sense when real users are asking questions you can't fully predict. At launch, you don't know the gaps in the agent's knowledge or process. Escalation turns those gaps into a visible queue. Trace analysis turns repeated failures into concrete edits. The creator's argument is that production traces are the most valuable data you have, and that most teams don't use them.

The cost is human attention, especially early on. The expected pattern is that escalations drop as the knowledge base fills in. If your inbox never shrinks, that is a signal that the lessons aren't being retrieved well or the core instructions need work.

One caveat: the demo used a simple dataset and a demo version of a real agent, with a custom dashboard. The 10-second timings and the clean generalization come from a controlled example, not a measured production benchmark. Test with your own traffic before trusting the numbers.