--- title: "How to Build AI Agent Evals: Error Analysis, Graders, Judges" url: "https://goremy.ai/blog/articles/how-to-build-ai-agent-evals" author: Luis Chavez-Mattos editor: Luis Chavez-Mattos editor_title: Director, Product published: "2026-11-10T13:00:08.976Z" updated: "2026-11-10T16:57:06.248Z" topic: AI Agents & Automation description: "Build your first AI agent eval set with a worked SQL agent example: product vs model evals, code graders, LLM judges, human labels, and error analysis." --- # How to Build AI Agent Evals: Error Analysis, Graders, Judges > Build your first AI agent eval set with a worked SQL agent example: product vs model evals, code graders, LLM judges, human labels, and error analysis. Edited by Luis Chavez-Mattos, Director, Product ยท Published 2026-11-10 ## How do you build evals for an AI agent? Build a set of questions your real users would ask, run each one several times, and grade the answers on properties like correctness and rule-following instead of exact text. Start by reading real outputs by hand, then add code checks for anything exact, then add an LLM judge for what remains. Calibrate the judge against human labels, and rerun the full set after every change to a prompt, tool, or model. ## TL;DR - Agents sample their output, so you must **run every test several times** and track how often the agent gets it right, not whether it did once. - **Product evals beat public benchmarks** for deciding if your agent works, because benchmarks know nothing about your business rules. - Use three graders in this order: **human reading first, code checks second, LLM judge third**. - **Error analysis** (read real answers, note the first thing that went wrong, group and count) tells you what to test. - **Check the judge against human labels.** The disagreements often expose bad labels, which improves both sides. - **Rerun the entire eval set after each fix**, because a change that fixes one failure can cause another. ## Why is testing an agent different from testing normal code? A unit test works because a function does the same thing every time. An agent does not. Two things break that assumption. First, the model samples its output. Ask the same question three times and the agent can take three different paths through the database and return three different answers. Second, there is rarely one correct string. A revenue question can be answered with a sentence, a table, or a number plus a short explanation, and all of them can be right. So you grade properties: Is the number right? Did the agent follow the business rules? Did it make something up? And because results vary, you run each test multiple times. The person asking about revenue gets one answer. An agent that is right most of the time can still hand someone a wrong number on a given day. ## What is the difference between model evals and product evals? A model eval is a public benchmark: a fixed set of questions every lab runs so models can be compared. Some are saturated. On MMLU Pro, a general knowledge benchmark, top models sit so close to the maximum that one major leaderboard stopped using it. Others are task-specific. Bird, for example, tests how well a model turns questions into SQL. These help you pick a model that is good at a task in general. The catch is that a benchmark only knows its own databases. It has no idea your fiscal year starts in February, that three customers are test accounts, or that a product was renamed last year. A product eval is a set of questions your own users would ask, graded by your business's rules. Use benchmarks to choose the model. Use product evals to check whether your agent does the job the way your business needs it done. ## What does a worked example look like? The example here is a small SQL data agent for a made-up coffee wholesaler, Amber Roasters. Its database has customers, products, orders, and refunds. The finance team has unwritten rules: revenue is counted after refunds, the fiscal year starts February 1, and three test accounts must never be counted. The agent ran on DeepSeek V4.1 Flash with two tools, one to list tables and one to run a query. The builder wrote 32 questions with correct SQL for each, split into two groups: | Group | What it needs | Result (code graded, 3 runs each) | |---|---|---| | Benchmark-style | Tables alone | About 9 in 10 correct | | Business-rule | A finance team rule | Fewer than half correct | Same model, same database, same prompt. Test only with generic questions and you would ship an agent that looks fine and isn't. ## What are the three types of graders, and in what order should you use them? You will end up using all three. 1. **Code.** A script checks the answer in a fixed way. For a SQL agent, run the agent's query and compare its rows with the rows from a query you know is correct. It is fast, cheap, and gives the same verdict every time. It cannot judge things like whether the agent stated its assumptions. 2. **LLM as a judge.** Another model reads the answer and grades it against a plain-English rubric. It handles fuzzy criteria, but it is a language model and can be wrong. Studies have shown judges tend to favor whichever answer they read first, especially smaller models. 3. **Human.** A developer or someone who knows the business labels answers pass or fail. It is the slowest option, but these labels are what you use to check both the code and the judge. The order matters. Many people start by writing a judge prompt, but you can't write a good rubric before you know how the agent fails. Start with a human reading real answers, write code checks for anything exact, and bring in a judge for what is left or ambiguous. ## How does error analysis work? Your hand-written questions only cover problems you already thought of. Error analysis finds the rest. Read a batch of real conversations, write down the first thing that went wrong in each, then group and count the notes. You learn which problems happen most and where to spend effort. In the example, the builder generated 60 questions phrased the way a sales team would type them, with typos and vague requests like "revenue month-to-date," then read every answer. Two findings stood out: - The biggest failure group was the **fiscal calendar**. Given "last quarter" or "year-to-date," the agent used calendar dates because nothing told it the year starts in February. - The second was **revenue**, where the agent reported sales before refunds. The failure the builder feared most never appeared. Eight questions asked about data the company doesn't have, such as sales targets or profit margins. Each time, the agent said the data was missing instead of inventing a number. Writing evals before reading answers would have aimed effort at the wrong problem. ## Can your grader be the thing that is broken? Yes, and it happened here. Asked which region made the most revenue, the agent named the right region, but the code grader also tried to match a revenue number nobody asked for. A correct answer was marked wrong. After the grader was fixed, about a quarter of the benchmark failures disappeared. Read your failures before you trust any score. Your grader is code you wrote, and it has bugs like any other code. ## How do you build and calibrate an LLM judge? Reading every answer works for dozens of conversations, not thousands a day. That is where a judge helps. The example went through two versions. **Naive judge.** It asked another model whether the answer was correct or helpful. It caught only 5 of the 26 labeled failures. A wrong number in a confident, well-formatted table looks correct and helpful to a model. **Specific judge.** The rubric came from the error analysis and checked four rules one at a time: fiscal calendar, revenue definition, test accounts, and handling of missing data. It caught almost every real failure but also failed some fine answers. The builder then reviewed every case where judge and human disagreed. In three, the judge was right and the human ground truth was wrong, including an answer that counted two test accounts as real customers. After tightening two rules and fixing the labels, the judge agreed with the labels on 90% of answers. ## Why rerun the whole eval set after every change? The fix for the example was to put the finance rules in the system prompt (version two). Business-rule questions went from under half correct to about eight in ten. Benchmark-style questions held steady overall, but the change fixed one and broke another. One question now returned no answer at all because the agent burned its step budget on tool calls. A fix aimed at one subset can hurt another place. Run the full set, multiple times, after every change to a prompt, tool, or model. Treat the eval set as a living thing too. Data drifts and user behavior changes, so the set has to evolve. ## What are the limits of this approach? - **Same-model judging.** In the example the judge ran on the same model as the agent, which may go easy on answers in its own style. Using a different model as judge is the safer choice. - **Sample size.** People who teach error analysis recommend reading a large batch of conversations (the source cuts off at "at least 100"), and a 60-question sample is small by that standard. - **Coverage.** Your set only reflects the questions you generated or collected, so keep adding real user traffic. ## Frequently Asked Questions ### Do I still need benchmarks if I have product evals? Yes, for picking a model. Benchmarks like Bird tell you whether a model is generally good at SQL. They cannot tell you whether it follows your company's fiscal calendar or excludes your test accounts. ### Should I write the LLM judge prompt first? No. You can't write a useful rubric until you know how your agent fails. Read real answers, build code checks for exact criteria, and only then write a judge for what remains. ### Why run each test more than once? The model samples its output, so the same question can produce different paths and answers. Repeated runs show how often the agent is right, which is what a user experiences on any given day. ### How do I know if my judge can be trusted? Compare its verdicts to human labels and investigate every disagreement. In the example, a naive judge caught 5 of 26 failures, while a rubric built from error analysis reached 90% agreement after tightening rules and correcting bad labels. ### What if my grader marks correct answers as wrong? Read the failures before trusting the score. In the example, fixing a grader bug removed about a quarter of the apparent benchmark failures.