AI agent evaluation: test sets, scoring and the pass bar before go-live

To evaluate an AI agent, build a test set from your own past cases, run the agent on them, and score two things: whether it reached the right outcome and whether it used its tools correctly on the way. Then agree a pass bar with the process owner before the agent touches anything live. A demo of ten good examples isn't an evaluation, and public benchmarks say little about how an agent will handle your invoices or tickets.

#Why agents need more than answer checks

Testing a chatbot means reading its answers. Testing an agent means checking what it did. An agent that handles a damaged-order ticket might look up the order, check the payment record, draft a reply and create a refund request. The final message can read well while the refund request names the wrong order.

Agents are harder to test than single model calls for a few reasons. They take many steps, and a mistake in step two can look like a success at step six. Two runs on the same input can take different paths, so one pass proves little. And their errors have side effects: a changed record, an email sent, a payment queued. A hallucination in a chatbot is a bad sentence. In an agent it can be a bad PO number written to your ERP.

Anthropic's engineering team makes a similar point in its guide to evaluating agents (opens in a new tab). Run several trials per task because outputs vary, and read the transcripts to learn whether a failure was a real mistake or a grader rejecting a valid answer.

Evaluation happens before launch. Runtime limits such as permissions and approvals are covered in our piece on AI guardrails. Evals show whether the agent behaves, and guardrails limit the damage when it doesn't.

#What to score

Score five things separately, so a failure points to a fix.

Anything exact belongs to code graders: field matches, totals, allowed values, forbidden actions. They're cheap, repeatable and objective. Use model graders with a written rubric for free text such as drafts and summaries, and check them against human grades before trusting them. Keep people for calibration and for the cases that matter most.

#Build the test set from real cases

The set is the asset. It outlives any model or prompt, and you'll rerun it for years.

1. Pull 100 to 300 real cases from your history: tickets with their final resolution, invoices with the posted ERP values, emails with the team that handled them. Use the original wording, typos included.
2. Have a process owner record the correct outcome for each, and the evidence behind it. For an exception agent, that means the right queue, the reason and the next action.
3. Add the hard cases on purpose: missing fields, skewed or rotated scans, a second language, duplicate submissions, requests just over a limit.
4. Add cases the agent should refuse or hand to a person, such as a bank detail change, a legal threat or a request outside its remit.
5. Add adversarial cases. An email that tells the agent to ignore its rules, or an attachment with instructions hidden in the text, checks whether the agent can be talked out of its limits.
6. Hold back a portion that is never used while you tune prompts. Otherwise you'll optimize to the set and overestimate quality.
7. Label each case with its type and risk, so you can report results by category rather than as one blended number.

Keep personal data out of the set where you can, or work in an environment cleared for it. Record where every case came from, so an auditor can see the set reflects real work.

#Run each case more than once

An agent that passes once may fail on the next run. Run each case several times and track consistency as well as accuracy.

The arithmetic is simple. If an agent gets a case right 90% of the time, the chance it gets the same case right three times in a row is about 73%. For a customer-facing or money-touching agent, "usually right" is a weaker standard than it sounds. Report two numbers: how often it succeeds at least once, and how often it succeeds every time. Use the second for anything that acts without a person.

Read the transcripts too. Numbers tell you something failed. The transcript tells you why: the agent misread a document, invented a value, called the wrong tool, or was marked wrong by a grader that didn't anticipate a valid alternative. Reading 30 failures in an afternoon teaches you more than any dashboard.

#Public benchmarks and what they miss

Published agent benchmarks measure general capability on public tasks, and they help when you're picking between models. They don't measure your approval limits, your chart of accounts or your supplier quirks. Use a leaderboard to build a shortlist, then compare the shortlisted models on your own test set. Only that result predicts production behavior.

#The pass bar for moving from shadow mode to live

Launch in stages, with a written bar at each. The numbers below are examples to show the structure. Your process owner should set the real ones, based on what an error costs.

1. Offline eval. The agent passes your test set at the agreed rate, with zero failures in the "must never happen" category, such as releasing a payment or contacting an unapproved domain.
2. Shadow mode. The agent runs on live cases but takes no action. A person handles each case as today, and you compare. Run it long enough to see month-end spikes or a seasonal pattern.
3. Assisted mode. The agent proposes and a person approves every action. Track how often people accept, edit or reject.
4. Limited live. The agent acts alone on the low-risk, high-confidence slice, for example invoices under a set amount from established vendors. Everything else still goes to review.
5. Widen in steps, only while the rejection and override rates hold.

A bar might read: at least 95% correct outcomes on routine cases, 100% correct handling of the refuse-or-escalate set, and no forbidden tool calls across all trials. Where you set it depends on the cost of a wrong action. A mis-tagged support ticket and a wrong bank detail are different risks.

Key point: Build the test set from your own history, score outcomes and tool calls separately, run each case more than once, and write the pass bar for each launch stage before you see the results.

#Keep the evals running after launch

Models change, prompts get edited and your business drifts. Treat the test set as a regression suite and rerun it whenever the model version, the instructions, the tools or the rules change. Feed real production failures back into it, using reviewer overrides as a source. Watch the rate of human corrections over time, because a slow rise is the earliest sign of drift. Name an owner for the set. Without one it goes stale like any other documentation.

Approvals need design too. Our notes on human-in-the-loop AI cover how to build review steps people don't rubber-stamp, and what an AI agent is explains the parts being tested. For the glossary definition of evals, the short version is a repeatable test of AI behavior against known answers.

#Common mistakes

• Testing only the happy path, so the first strange invoice is also the first failure.
• Grading the final message and ignoring the tool calls that produced it.
• Reporting one blended accuracy figure that hides a weak category.
• Letting the build team grade its own work with no process-owner sign-off.
• Tuning against the whole set and having nothing left to test on.
• Skipping shadow mode because the demo looked good.

We build these test sets and launch gates into every AI agents project. If you have an agent in a pilot and want an independent view of whether it's ready, book a strategy call. Bring a sample of past cases and we'll show you what a test set and pass bar would look like for your process.

#Frequently asked questions

#How do you evaluate an AI agent?

Run it on a test set built from real past cases and score outcomes and tool calls separately. Repeat each case several times to measure consistency. Compare results with a pass bar the process owner agreed in advance.

#What is an eval set?

It's a collection of real and deliberately hard cases with the correct outcome recorded for each. You run the agent against it every time something changes, so you can see whether quality went up or down. A good one includes cases the agent should refuse.

#How many test cases do we need?

Start with 100 to 300 real cases, with extra weight on hard and high-risk ones. Coverage matters more than count: every category, language and exception type you handle should appear. Grow the set from production failures.

#Can another AI model grade the agent's work?

Yes, for free-text quality, using a written rubric. Check its grades against a sample graded by people before relying on it, and use code-based checks for anything exact like totals, fields and forbidden actions.

#When is an agent ready to go live?

When it meets the agreed bar on the test set, has run in shadow mode on live cases without surprises, and passes human review in assisted mode. Go live first on the low-risk slice with approvals still in place for anything costly.