Build AI evaluations from real tasks, define expected outcomes, and test critical failures. Use the results to guide release and ongoing review.
Most agent projects reach a moment when the demo works. Someone asks a question, the agent pulls the right record, calls the right tool, and writes a clean answer. The room relaxes. That moment is useful, but it proves less than it seems. A demo shows that the system can succeed. It does not show how often it succeeds, where it fails, or what a failure costs.
Before launch, test an agent against work it will actually receive. An evaluation set, or eval set, pairs inputs with expected outcomes. Real cases should form its base. Add synthetic cases for rare failures, attacks, and situations that need controlled coverage.
Evaluations make prompt, model, and tool changes easier to compare. Passing a test set supports a release decision; it does not replace permission checks, monitoring, or human review.
An original diagram of an eval harness built from real tasks.
Collect the work, not the idea of the work
Go where the tasks already live
The best eval cases already exist. They sit in support tickets, shared inboxes, call transcripts, CRM notes, document folders, and application logs. If an agent will answer billing questions, pull a few hundred recent billing tickets. If it will summarize contracts, gather the contracts the legal team actually reviewed last quarter, along with the notes they wrote.
Real cases carry details that invented ones miss: typos, partial information, attachments in the wrong format, customers who ask two questions at once, and references to a policy that changed last month. Those details are where agents fail, so they belong in the set.
Mask or remove personal data before cases enter the set. An eval set gets copied, shared, and run hundreds of times. Treat it like any other dataset that started with customer information in it.
Include the boring, the hard, and the wrong
A useful set has a deliberate mix:
- Routine cases. The high-volume, unremarkable requests that make up most of the work. If the agent is shaky here, nothing else matters.
- Hard cases. Ambiguous inputs, long documents, multi-step requests, and questions that need several tool calls to answer.
- Counterexamples. Cases where the right answer is to refuse, ask a clarifying question, or hand off to a person. A refund above the agent’s limit. A question that needs a licensed professional. A request that would require account access the agent should not have.
Counterexamples are the part teams skip most often, and they matter most. An agent that answers everything confidently will score well on a set that only contains answerable questions. It will also cause the incidents that end projects.
Start with fifty to a hundred cases, not thousands. A small set the team has read carefully is worth more than a large one nobody has looked at.
Define what good means for each task
Write the expected outcome down
Every case needs an expected outcome, written before the agent runs. For some tasks it is a single value: the invoice total, the ticket category, the account ID. For others it is a set of properties: the reply cites the current refund policy, offers the correct next step, and does not promise a delivery date.
Writing these down is where much of the real product thinking happens. Two people who agree that the agent “should handle refunds” often disagree about the right answer for a specific refund. It is better to find that out in a spreadsheet than in production.
Separate outcome from style
Be explicit about what is required and what is preference. A support reply that gives the right answer in a slightly different tone is a pass. A reply that sounds perfect and quotes last year’s policy is a fail. Mixing the two makes scores noisy and hides the failures that matter.
Match the grader to the question
Use the simplest grader that is reliable
Not every case needs the same kind of judgment. A practical harness uses several grader types, and each case uses the simplest one that gives a trustworthy answer.
- Deterministic checks. Exact match, numeric tolerance, a pattern match, or a lookup against a known answer. Fast, cheap, and unambiguous. Use them whenever the expected outcome is a value.
- Schema and tool-call checks. Did the output parse against the expected schema? Did the agent call the right tool with valid arguments, in a sensible order, without touching tools it should not use? For agents, this is often where the real failures show up.
- Rubric-based model graders. A model scores the output against a written rubric. Does the reply cite the correct policy? Does it stay in scope? Does it escalate when it should? Useful for open-ended outputs where no single correct string exists.
- Human review. For cases where judgment is the whole point, or where the stakes are high enough that nobody should trust an automated score alone.
Calibrate model graders against people
A model grader is itself a system that can be wrong. Before trusting one, have a domain expert label a sample of outputs, run the grader on the same sample, and compare. Where they disagree, tighten the rubric or split it into narrower yes-or-no questions. Repeat the comparison from time to time, because a grader drifts when its prompt or its underlying model changes.
A rubric grader that usually agrees with your experts is a useful instrument. One that has never been checked is an opinion with a number attached.
Measure more than accuracy
Track the operating numbers
A correct answer that takes forty seconds and costs more than the person it was meant to help is not a success. Each eval run should report:
- Task success rate, broken down by case category, not only in aggregate.
- Cost per task, including every model call, retrieval step, and retry.
- Latency at p50 and p95. The median shows what a typical user sees. The p95 shows what the unlucky ones see.
- Escalation rate, and whether escalations happened on the cases that should escalate.
- Tool errors, such as failed calls, invalid arguments, and timeouts.
Don’t let the average hide the failure
A strong headline score can conceal a category where the agent fails every time. If that category is “requests to cancel a contract,” the average is irrelevant. Report results per category, and flag any failure on a critical case separately, regardless of the overall number.
Mark some cases as critical. A single failure on one of them blocks a release, no matter how good everything else looks.
Agree on the bar before launch
Set thresholds with the business
The team building the agent should not decide alone what “good enough” means. Before launch, sit down with the people who own the workflow and agree on thresholds: a minimum success rate per category, zero failures on critical cases, a maximum cost per task, a p95 latency ceiling, and an acceptable escalation rate.
Writing these down in advance changes the launch conversation. Instead of debating whether the agent feels ready, the team can point to a scorecard and say which thresholds pass and which do not.
Run the suite on every change
Once the set exists, it becomes a regression suite. Run it on every prompt edit, model upgrade, tool change, and retrieval change. Prompts behave like code, and a one-line tweak that fixes one case can quietly break five others. Without a suite, nobody notices until users do.
Wire the run into the place where changes are already reviewed, so a pull request that edits a prompt shows its scorecard next to the diff.
Keep the set alive after launch
Let production write new cases
Launch is when the eval set starts to get good. Production traces show which requests the agent sees most, where it hesitates, and where people step in. A review queue, where a person checks a sample of outputs plus anything flagged as low confidence, turns those observations into labeled cases.
The loop is simple. A reviewed failure becomes a new case with an expected outcome. The case enters the set. The next change has to pass it. Over a few months, the set comes to reflect the real distribution of work, including the edge cases nobody imagined at the start.
Watch for the common traps
The same few mistakes show up again and again:
- Testing on the author’s examples. The person who wrote the prompt tested it on cases they had in mind. Those cases may favor the prompt. Include real cases that were not used while developing it.
- Averaging away the failures that matter. One headline number, no categories, and no critical cases.
- Graders that drift. A model grader calibrated once and never rechecked, quietly scoring against a standard nobody agreed to.
- Eval sets that never grow. A set frozen at launch slowly stops describing the work. Scores hold steady while quality in production slips.
A practical starting point
Build a small first set
You do not need a platform to start. Pick one workflow the agent will handle, then:
- Pull fifty to a hundred recent real cases from tickets, documents, transcripts, or logs, and mask personal data.
- Tag each case as routine, hard, or counterexample, and mark the critical ones.
- Write the expected outcome for each, reviewed by someone who does the work today.
- Add deterministic and tool-call checks first, then a rubric grader for the rest, calibrated against a labeled sample.
- Agree on thresholds with the workflow owner.
- Run the suite on every change, and add a case for every reviewed failure after launch.
A spreadsheet and a script are enough for the first version. The discipline matters more than the tooling.
Keep a separate set of cases for final validation. Run nondeterministic tasks more than once when a single pass would hide variability. After launch, use reviewed failures to extend the set.