How Do You Stop Deferring to Your Coding Agent's Plans?
Anthropic's study found analysts followed their agent's lean 86% of the time. Here are three planning fixes for Claude Code and Codex.
How do you stop deferring to your coding agent when planning?
Change the structure of the planning session so the agent's top recommendation is not the default path. Three habits help. Settle your priorities before the agent proposes any solutions. Build cheap, throwaway prototypes to resolve close decisions with evidence instead of argument. Ask the agent what would change its advice, so you see the conditions where its pick fails. None of these remove the pull toward agreeing, but they force you to supply judgment instead of assuming you did.
TL;DR
- Anthropic's study found analysts went with their agent's lean 86% of the time, where a coin flip would give roughly 50%.
- People who already had a preference switched away from it 60% of the time when the agent disagreed, versus 6% when it agreed.
- Afterward, 77% of analysts reused their agent's arguments to explain their choices, but only 15% credited the agent as the source.
- Asking about priorities before solutions gives you a standard to judge recommendations against.
- A throwaway prototype turns a debate about which option is better into something you can look at and run.
- Asking "what would change your advice?" exposes the assumptions behind the agent's top pick.
What did Anthropic's study actually find?
Anthropic ran an experiment on expert analysts working with AI agents. Each analyst's agent was secretly set up, through its system prompt, to favor one option on each decision. Which way it leaned was randomized. The agent was told to argue its case but could change its mind if given good enough evidence.
The results:
| Finding | Number |
|---|---|
| Analysts who went with their agent's lean across four analytical choices | 86% |
| Expected rate if the agent's input made no difference | about 50% |
| Switched away from their own earlier preference when the agent disagreed | 60% |
| Switched away when the agent agreed | 6% |
| Used their agent's arguments when explaining their choices | 77% |
| Credited the agent as the source of those arguments | 15% |
| Called the final recommendation more their own than the agent's | 52% |
Two other takeaways matter. Experience and expertise did not determine who deferred. And analysts who worked without AI made more serious errors and took longer, so the study does not argue for using agents less. Anthropic says used well, AI can expand agency instead of eroding it.
The caveats are real. Anthropic deliberately strengthened the agent's lean. It studied one task, in one field, with one model. Still, the pattern will look familiar to anyone who has watched themselves click the first multiple-choice option for forty questions in a row.
Why does this hit coding agent planning so hard?
Most of us plan the same way. We describe what we want at a high level, then the agent interviews us, often through a multiple-choice question tool. The agent usually marks its recommendation as the top option. Answering is fast, and it feels like decision-making.
Skills like Matt Pocock's Grill Me exist to pull intent out of you, and he names the risk in his docs: "The failure mode is passivity, answering agreed, agreed, agreed for 40 questions, coming out with a plan the agent wrote and you nodded at."
There is also a catch-22. Agents are most useful when your idea is vague and you want help turning it into requirements. That is exactly when your views are least formed, and when deferring is easiest. The study suggests experts often don't arrive with fully formed views, which makes them more likely to adopt the agent's.
How do you ask about priorities before solutions?
Most bad deferral starts at the wrong level. If the agent opens with "Should we use a queue or a cron job?", you are judging two solutions without a stated standard. The agent's pick wins because it sounds reasonable.
Reverse the order. Before any implementation question, have the agent ask what matters for this project, then write the answers down. Examples of what to settle:
- How long will this code live? A weekend experiment and a core system need different answers.
- What is the cost of being wrong? Data loss, downtime, and a slightly ugly UI are not equivalent.
- What are you optimizing for: speed to ship, ease of maintenance, performance, or low cost?
- What can the team actually operate and debug later?
Then put those priorities in the planning context. When the agent recommends something, you can ask which priority it serves and which it sacrifices. A recommendation is now an argument you can check against your own criteria. You have also done the part of the thinking that is hardest to outsource: deciding what "good" means here.
Can you prototype instead of debating?
Planning arguments feel decisive because they are articulate. They are not evidence. When two options are close, and the agent's confidence is the main thing separating them, stop talking and build both.
Agents make this cheap. Ask for a minimal version of each approach, in separate branches or directories, with no polish and no intent to keep either one. Run them. Look at the code size, the awkward spots, the failure behavior, and how each one feels to extend.
A few rules keep this useful:
- Set a small time or token budget for each prototype so it stays disposable.
- Run the prototypes in parallel, using separate agent sessions or worktrees, so one does not bias the other.
- Define what you are comparing before you look at results, tied to the priorities you set earlier.
- Delete the losers. The goal is a decision, not code.
This shifts your role from reviewing prose to reviewing behavior. It is harder to nod along with a recommendation when you have two working samples in front of you and one of them is clearly clumsy.
What happens when you ask what would change its advice?
Anthropic configured its experimental agents to argue a position but update on good evidence. Real agents are similar: they commit to a recommendation, and they will usually revise it if you give them a reason. Most people never ask for the reason it might be wrong.
After the agent recommends an option, try prompts like:
- "What would have to be true for your second choice to be better?"
- "What assumptions does your recommendation depend on? Which are you least sure of?"
- "Argue against your own recommendation as strongly as you can."
- "What information from me would change your advice?"
The last one is especially useful. It turns the agent's answer into a list of questions you can actually verify: your expected scale, your team's skills, your deployment constraints. If the agent says its pick depends on traffic staying low and you know it won't, you have found a real disagreement from your own knowledge instead of from a vague feeling.
Treat the devil's advocate output with the same skepticism as the original. A model can argue either side fluently. The value is in surfacing assumptions you can test, not in finding a new winner.
Does using a different agent for review fix this?
Only partly. The study raises a broader concern: if separate teams all defer to a model with the same preferences, their agreement may not be independent. The engineering version is familiar. One person has an agent write a pull request, another has an agent review it. Different skills and prompts help, but the same underlying model is doing both the building and the checking.
So pair these habits with independent inputs. Your own written priorities, real prototypes, and human reviewers who form opinions before reading the agent's summary all add signal that does not trace back to one model's prior beliefs.
Is this a solved problem?
No. Anthropic's article ends with open questions: how to measure deferral, how to design agents that ask better questions and express appropriate uncertainty, and how institutions should treat expert sign-off when AI does more of the work. The workflow fixes above are practical mitigations, not guarantees. They work by adding friction and independent evidence at the points where agreement is cheapest.
Frequently Asked Questions
Why do coding agents put their recommendation first in multiple-choice questions?
Planning skills and question tools often have the agent mark its preferred answer, which makes accepting it the lowest-effort path. The study suggests low effort plus a confident agent is enough to steer most decisions. You can change the prompt so the agent lists options without ranking them until you have answered.
Does the Anthropic study prove I'm deferring too much?
No. It showed a strong effect in one task, one field, and one model, and the agent's lean was deliberately strengthened. It is best read as a warning that the effect can be large, not as a measurement of your own behavior. Reviewing your last few plans for decisions you can't justify without quoting the agent is a quick self-check.
Are prototypes worth the extra tokens and time?
For decisions that are expensive to reverse, usually yes. Minimal versions of two approaches cost little compared with rebuilding a wrong architecture later. For small, easily reversed choices, a quick priorities check or a "what would change your advice?" question is enough.
Should I use AI less for planning?
The study does not say that. Analysts working without AI made more serious errors and took longer. The aim is to keep the speed and error reduction while making sure the decisions that depend on your judgment are actually yours.
How we source, write, and correct our articles. Editorial standards
