Agents are non-deterministic, multi-step and opaque. Ordinary testing habits do not transfer cleanly, and the usual result is a system nobody can tell is getting worse.
Testing without exact matches
String comparison is useless here — two different, equally correct answers fail. Score outcomes instead.
Build a case set of 20–50 real tasks, each with criteria that can be judged:
Case: "Where is order 4471?"
- Called
get_order_by_idwith4471— yes/no- Stated the correct status — yes/no
- Did not invent a delivery date — yes/no
- Finished within 5 steps — yes/no
Now you have a score. Run each case several times, because a system that passes once in three is not passing.
Include the hard cases deliberately: ambiguous requests, missing data, tools that fail, and at least one input containing text that looks like an instruction.
Log enough to reconstruct
For every run, capture: the input, the full sequence of model decisions and their reasoning, every tool call with its arguments, every result including errors, timing per step, total tokens and cost, and the final outcome.
The test is whether you can answer "why did it do that?" three weeks later from logs alone. Anything less and you are guessing about a system you cannot reproduce.
Watch these in production
- Completion rate — how often runs reach a successful end
- Steps per run — a rising average means it is working harder for the same result
- Tool error rate — per tool; a spike usually means an API changed
- Escalation rate — how often it hands to a human, and why
- Cost per run — the first thing to move when something goes subtly wrong
- Runs hitting the iteration cap — each one is a task it could not finish
Alert on changes, not absolutes. A completion rate that fell from 94% to 78% overnight is the signal.
Regressions from underneath
This is the failure mode unique to building on someone else's model: your code did not change, and behaviour did. A provider updates a model, and a prompt tuned to the previous version behaves differently.
The defences are ordinary:
- Pin the model version where the provider allows it
- Re-run the evaluation set on a schedule, not only when you change something
- Keep a baseline so you can say what "normal" was
- Re-run before adopting a new version, and compare rather than assume
The honest position
You will not reach the reliability of deterministic software, and that is the correct expectation rather than a failure. The goal is knowing your actual rate, detecting when it moves, and containing the consequences when the system is wrong — which is what every control in this course has been building towards.