Why Does Your AI Coding Agent Say It's Done When It's Broken?

Why does my AI coding agent say it's done when it isn't?

Your coding agent reports success because it has no built-in reason to doubt itself. It writes the code, writes its own tests, sees them pass, and summarizes the work as finished. It never opens the app the way a user would, and nobody asks it to prove anything. The fix is to change what you require of it: ask it to argue against its own work, hand the review to a second agent with fresh context, and demand real evidence such as commands, output and screenshots before you accept "done."

What is the quickest fix for a false "done"?

Play devil's advocate in the same conversation. When the agent hands back its work, ask it to attack it. Agents are good at finding gaps when you ask them to look for gaps.

Copy this:

What would make this not actually work? What are the gaps between
what the issue asked for and what you did?

Expect a long list. The agent wants to please you, so it will often produce more concerns than you need. Don't act on all of them. Skim for the two or three items that make you think "yes, that is definitely missing." Those are usually the real holes, and you would have found them yourself ten minutes later, after wasting a review cycle.

This takes seconds and needs no setup. Its weakness is that the same agent that built the thing is now judging it, and it carries all its earlier assumptions along.

How do I set up a reviewer skill with fresh context?

A "skill" is a saved set of instructions your coding agent can load for a specific job. A reviewer skill is a skill for reviewing. You hand it to a different agent session that has not seen the building process, so it looks at the result with no loyalty to the decisions that produced it. It can also send its findings back to the builder to fix.

Many agent frameworks ship a version of this, which says something about how well the pattern works. But a generic reviewer checks generic things. What you want reviewed depends on what you are building. A reviewer for a customer-facing web form should care about validation and error messages. A reviewer for a data import should care about bad rows and duplicates. Writing your own version lets you encode that.

A starting point you can adapt:

You are a reviewer. You did not build this. Read the original request
and the changes made. List every place where the result does not meet
the request. Check edge cases, empty states, and error handling.
For each gap, say what is missing and how to confirm it. Do not
praise the work. Send the gaps back to the builder.

You do not need to write this alone. The approach that works is to describe the problem to your coding agent in plain words and let it help build the skill:

You keep telling me things are done, but I find gaps when I look
myself. I want a separate reviewer agent that reviews with fresh
context, the way I play devil's advocate. Where in my skills, rules
or setup should this live? Propose the change and we'll make it together.

Once it is saved, every future session can use it after an implementation.

How do I make my agent prove the work actually runs?

By default, agents give you a summary of what they built and a statement that everything works. That leaves you to verify it all yourself. Change the default so every claim arrives with evidence.

Good evidence is specific:

For full-stack apps, one approach is to have the agent use a browser command-line tool such as the agent browser CLI. It can open the running app, perform the feature it just built, capture real screenshots, and attach them to the pull request body. You review the pull request the way you would review a colleague's work, with proof in front of you. Reviews get faster, and you trust the agent more because it shows its work.

One detail worth copying: put the evidence plan in your planning step. Before the agent writes any code, have it list the testing artifacts it will produce. Then you know in advance what "finished" will look like, and the agent cannot quietly substitute a lighter standard later.

Prompt to try:

Every claim you make needs evidence. When you finish, show me the
commands you ran with their output, and screenshots of the feature
working in the real app. Where should this requirement live in my
rules or skills? Should it be its own skill?

Why do tests pass when the app is still broken?

Because the agent tested the code and never used the product. Coding agents write plenty of unit and integration tests, and the volume looks reassuring. But a pile of passing tests can still miss a button that does nothing or a page that fails to load. The creator whose workflow this article draws on goes further and says validation is among the things coding agents are worst at, even when you run it in a separate session.

The fix is to test the way a user would. Use a browser automation tool so the agent opens the real app, clicks buttons, types input, and looks at what happens. There are two ways to wire that in, and both work.

Option 1: change your planning skill. Add a validation step to planning, so the agent decides up front how it will exercise the app and what it will capture.

Option 2: create a "try it" skill. This is a small, dedicated skill with one rule: after any change a user can see, open the real app with the browser tool and test it as a user would.

The tests you write for yourself all pass, but the app is still
broken. I want you to make a skill called "try it". After a change
to anything a user can see, open the real app with the agent browser
CLI and test it the way a user would. Show me what you did.

These are the same solution in different places. Which is better depends on you. If you already lean on a detailed planning step, extend it. If you want a single reusable habit that fires after every visible change, make the skill.

Where should each fix live: rules, skills or hooks?

Getting this wrong is why some fixes don't stick. Your coding agent's setup has three main parts, and each has a job.

Hooks attach a small piece of code to events, such as every time the agent is about to use a tool or when a conversation starts. A classic example is blocking the agent from reading your secrets file. You can write "don't read the .env file" in your rules, but the agent may still try. A hook stops the action even if it tries. Most AI coding assistants support hooks.

For the false-"done" problem, the reviewer and "try it" skills are workflows, so they are skills. If you find yourself writing hard requirements into your rules that keep getting ignored, ask your agent to scan your rules file for anything shaped like "when doing X, never do Y" and turn those into hooks.

How do I change my setup without guessing?

Use the same short conversation for every fix:

1. Describe the problem in plain words, as you would to a colleague.
2. Offer a suggested fix, such as a reviewer agent or a try-it skill.
3. Ask where it belongs in your skills, rules or hooks.
4. Approve the change and let the agent edit the files.

The changes then apply to every future session. Because you are shaping them yourself, they fit your projects and your tools. Not every problem will apply to you, and that is fine. Pick one, such as the devil's advocate prompt, and add the others as they bite.

None of this guarantees a flawless result. It moves you most of the way, and it moves the checking from your afternoon to the agent's workflow.

What does this mean for you?

If you build with a coding agent, even occasionally, you are the last line of defense against confident but wrong output. That is tiring. The fixes above shift some of that load back.

• If you are new to this: start with the devil's advocate prompt. It costs nothing and catches real gaps today.
• If you review pull requests from an agent: ask for screenshots and command output in the pull request body. Reject "it works" without proof.
• If you manage people who use these tools: agree on what counts as evidence, so "done" means the same thing for everyone.
• If you've been burned by a broken release: add a try-it step so the agent opens the real app before it reports back.

Is there a way to get browser testing without building it myself?

If you would rather not assemble a review-and-evidence setup, Remy builds this checking into the process. It is an AI agent that builds and deploys full-stack business apps from a plain-English description. Behind each build is a QA agent that drives a real browser, clicks through the flows, checks desktop and mobile layouts, and records walkthroughs, so you see the app working instead of reading a claim. That suits a small internal tool, such as a request form or tracker, that you want to own instead of patching together with spreadsheets.