In the chapters so far, we've designed agents that hold tools, use memory, and divide work across several. Here is a wall many developers hit: "I built it, but I can't tell whether it's actually working properly." Because agents run probabilistically, the same instruction doesn't produce exactly the same behavior each time. That's precisely why "a way to measure (evaluation)" and "a way to make things visible (observability)" become the lifeline of putting one into real use. In this chapter, we build the foundation for grasping quality in numbers, reproducing failures, and continuously turning improvements.
The goal: "improve your agent with measured numbers, not gut feel"
Why evaluation matters most
An ordinary program returns the same result given the same input. But at the heart of an agent is an LLM, and its output is probabilistic. Even with the same instruction, the tool it picks, the steps, and the phrasing vary slightly each time. In other words, just because it worked once, there's no guarantee it'll work the next time.
Here lies a pitfall unique to agent development. It ran beautifully in the demo, then broke the moment a real user phrased the request a little differently — this is not rare. The reason is simple: "it worked" doesn't mean "it's correct." It may have just happened to land the right answer, and it may be wrong 3 times out of 10. The only way to know is to count.
💡 You can't improve what you can't measure. "It kind of feels better" is not improvement. When you changed the prompt, did the success rate go up or down? When you added one tool, did it get needlessly slower and more expensive? Only when you can compare in numbers can you judge whether a change is progress or regression. Evaluation isn't "homework after building" — it's the foundation of development.
Another important thing is not to put evaluation off. You don't need a perfect eval foundation from the start. But if you proceed as "build first, think about evaluation once it works," evaluation usually never gets built. Small as it may be, start measuring right after you build your first agent — the apparent detour is the fastest route. For the big picture of evaluation, A practical guide to AI agent evals is a starting point; for what it even refers to, What are agent evals.
What to evaluate — outcome and trajectory
Hear "evaluate" and most people picture only whether the final answer is correct. That's certainly the starting point, but for agents it's not enough — because with agents, "what it did along the way" matters a great deal. Even if it reaches the correct answer, if it called useless tools 10 times and wasted time and cost, it's unusable in practice. Conversely, sometimes the answer was regrettably wrong but the judgments along the way were sound.
So we think of the subject of evaluation in four aspects. Keep these four in mind and the hesitation of "I don't know what to measure" disappears.
Did it achieve the given goal? Is the answer accurate, does it meet the requirements? The most basic metric.
Which tools, in what order, and why. Is the process sound? Are there detours or unnecessary steps?
Did it call the right tool with the right arguments? Is it calling nonexistent tools or ignoring failures?
The tokens, price, and time consumed in one run. Is it in balance with the quality?
📊 Measuring the "trajectory" is the crux of agent evaluation. Where evaluating ordinary AI output is "checking the answer," agent evaluation looks at "the answer plus the process that got there." An agent with a messy process collapses easily when conditions shift a little. One with a coherent process can right itself even when things drift. Precisely because you look at the process, you find the clues for improvement.
Five evaluation methods
"So, how do you measure?" — this is the heart of practice. There are several kinds of evaluation methods, and each has different strengths and weaknesses. Rather than rely on one, the standard play is to combine them according to what you want to measure. Let's look at five representative ones, with pros and cons.
Match the agent's output against a "correct answer" prepared in advance and measure agreement. Suits tasks with a single fixed answer, like classification, extraction, or calculation.
Pro: clear, easy to automate, and doesn't waver.
Con: hard to use for free-form writing where "the answer isn't a single one."
Judge mechanically by fixed conditions — "does it contain a required term," "is it valid JSON," "are there no banned words."
Pro: fast and cheap, strong at checking format and safety conditions.
Con: can't measure fuzzy quality like "does the meaning fit."
Give another LLM grading criteria and have it evaluate the output's quality. Can measure tasks with no single answer, like the quality of a summary or the helpfulness of a reply.
Pro: can auto-evaluate free-form or subjective quality at scale.
Con: the grader itself wavers/biases. Needs explicit criteria and human spot-checks.
Gather representative tasks into a test set and run them all on every change. A mechanism that guards "whether what passed until yesterday broke with today's fix."
Pro: detects regressions (degradation) from changes early.
Con: building and maintaining the test set takes effort.
After release, continuously observe real users' runs. Track the trend of success rate, cost, and latency, and gather user ratings (👍/👎) to follow real-world performance.
Pro: discovers problems that only appear in the real environment and unexpected usage.
Con: after the fact. Problems are known only "after they happen." Premised on pairing with up-front evaluation.
⚠️ Combining is the norm. Format and safety conditions with rule-based, fixed-answer things with ground truth matching, free-form with LLM-as-judge, every change with regression testing, and after release with production monitoring. Don't try to cover everything with one method. Sift most of it with the cheap, fast methods (①②), and use the expensive method (③) only where needed, and you keep costs down while holding accuracy. For more, see What is LLM evaluation.
Building an eval dataset (test set)
Whatever method you use, the foundation is "the material for what you measure" = the eval dataset (test set). In short, it's a collection of representative tasks capturing "given this kind of input, we want this kind of result." If this is weak, no matter how splendid the evaluation method, it won't be reliable. Conversely, with a good test set alone, evaluation is 80% won.
Building it is simple; proceed in this order. Rather than aiming for perfection, the rule is to start small and grow it.
Collect requests the agent is likely to actually get, including typical, edge, and failure-prone examples.
Write "how it should turn out" for each task. It can be a correct answer, conditions to meet, or grading criteria.
10–20 is enough at first. Prioritize "quality" that reflects reality over quantity.
Add failure cases found in production to the test set. It becomes an asset that gets smarter the more you use it.
✅ Don't underrate "just 10." You may feel "the sample is too small to mean anything." But 10 carefully chosen cases are far more useful than 1,000 gathered aimlessly. Get 10 representative cases you're actually struggling with measurable first. That alone lets you judge on the spot whether a prompt or tool change is a good or bad direction. A test set isn't made once and done — it's a living asset you grow one case at a time each time you find a failure.
Observability — making behavior visible
If evaluation is "measuring good and bad," then observability is making it possible to "trace why it turned out that way." Even if evaluation tells you "the success rate dropped," you can't fix it unless you can see what was happening inside. Because an agent turns multiple steps internally, from the outside it tends to be a black box: "put in an input, something happened, out came a result." Opening that up so you can see inside is observability.
There are broadly three elements that support observability.
One run recorded end to end. You can follow the whole flow of "input → thought → tool call → result → next decision" in chronological order.
For each step, what prompt was sent, with what arguments a tool was called, and what came back. Token consumption and elapsed time are kept here too.
Being able to retrace a run that went wrong with the same input and same state. Only once you can reproduce it can you pinpoint the cause and verify the fix.
You can add these to logs by hand, but in practice it's common to use a tracing tool. A tracing tool automatically captures each step of the agent and visualizes it neatly as a chronological tree. "Which step took time," "where a tool call failed," "at which decision it veered off in a strange direction" — all become visible at a glance.
📊 A trace is "eyes for debugging." In the example above, the tool call fails at ④, and at ⑤ the agent itself recovers. Look only at the result and it's "success," but look at the trace and you see the weakness that "it's prone to getting the column number wrong." This insight leads to the next improvement. The idea of observability is explained systematically in What is AI observability.
Turning the improvement loop
The toolkit so far — evaluation and the test set, plus observability — is useful on its own, but its true value emerges when turned as a single loop. Agent development isn't "build once and it's done"; it's the work of getting a little better through the cycle of measure, find the weakness, fix, and measure again.
Run the evaluation against the test set and produce success rate, cost, and latency in numbers.
Read the traces of failed runs and pin down where and why it went wrong.
Change one thing that addresses the cause — prompt, tool description, procedure, or division of labor.
Re-evaluate with the same test set and confirm the intended improvement appeared and nothing else regressed.
There's one important knack in this loop: change as few things at once as possible — ideally one. Change the prompt, the tools, and the procedure all at once, and even if the success rate rises, you won't know "which change did it." Precisely because you change one at a time and measure, cause and effect connect between change and result, and knowledge accumulates. It's unglamorous, but this is the line between "vague improvement" and "certain improvement."
✅ Turn the loop fast and small. Rather than landing one big overhaul, turning small changes quickly and repeatedly gets you better faster in the end. With 10 cases in the test set, you can measure right after a change. Once you internalize this "change → measure immediately" rhythm, agent development stops being a guessing game and becomes an engineering task that advances steadily.
- Agents are probabilistic, and "it worked" doesn't mean "it's correct." You can't improve without measuring.
- The subject of evaluation has four aspects — final outcome, the trajectory in between, tool calls, and cost and latency. The "trajectory" especially is the crux of agent evaluation.
- There are five methods — ground truth matching, rule-based, LLM-as-judge, regression testing, production monitoring — combined by subject.
- Build the eval dataset from a few representative tasks. Ten careful cases beat 1,000 aimless ones. Grow it by adding failures.
- Observability is execution traces, per-step logs, and reproducing failures. Make behavior visible with a tracing tool.
- Improvement is the loop of measure → find weakness → fix → measure again. Change one thing at a time; turn it small and fast.
Once you can measure quality and make behavior visible, the next question is "how to prevent runaways, misuse, and abuse." Let's learn the design for safely handling the autonomy we scaled up in the previous chapter, Chapter 4, "Multi-agent design." We move on to the next chapter, Chapter 6, "Guardrails and safety."