Evals First: How to Know Your AI Agent Actually Works

By Nam Bok Rodriguez · October 16, 2026 · 6 min read

You know an AI agent works when it passes a fixed set of real tasks, graded the same way every time, across several runs, and keeps passing them after every change. That set of tasks and graders is called an eval suite. We build it before the agent, because without it every prompt tweak or model swap is a guess.

Here is how we approach evals for agents, based on what the model providers publish and what holds up in practice.

What exactly is an eval for an agent?

Anthropic's guide Demystifying evals for AI agents, published in January 2026, gives a vocabulary worth adopting:

Task: a single test with defined inputs and success criteria.
Trial: one attempt at a task. You run several, because outputs vary between runs.
Grader: logic that scores some aspect of the agent's performance.
Transcript: the full record of a trial, every message and tool call.
Outcome: the final state of the environment when the trial ends.

That last point is the important difference from testing a chatbot. An agent changes things: files, tickets, records. So the most reliable check is often the state of the world afterward, not the wording of the reply.

Why isn't trying it a few times enough?

Because agents are not deterministic, and a handful of good demos tells you very little about the next thousand runs. OpenAI's evaluation best practices name this anti pattern directly: vibe based evaluation, where "it seems like it's working" stands in for a real test.

The Anthropic guide also separates two measures that are easy to confuse:

pass@k is the chance the agent gets at least one correct result in k attempts. Useful when a person can pick the best of several drafts.
pass^k is the chance that all k attempts succeed. This is the one that matters for customer facing agents, where every run must be right.

The gap grows fast. If an agent succeeds on a task 90% of the time and runs are independent, the chance it succeeds three times in a row is 0.9 cubed, about 73%. A demo shows you pass@1 on a good day. Your customers experience something closer to pass^k.

What should the first eval set look like?

Small and real beats large and synthetic. Anthropic suggests that 20 to 50 simple tasks drawn from real failures is a great start, and that converting user reported failures into test cases keeps the suite tied to actual usage.

Use real inputs. Pull from support tickets, past requests, or logs. OpenAI recommends mixing production data, expert written examples and synthetic cases that cover typical, edge and adversarial inputs.
Write a reference answer or end state for every task, so a grader has something concrete to check.
Balance the set. Test cases where a behavior should happen and cases where it should not. A suite that only checks "does it refund when asked" will happily reward an agent that refunds everyone.
Treat a 0% task as suspect. The guide notes that a task nobody passes across many trials usually signals a broken task, not an incapable agent.

How should you grade the results?

There are three kinds of graders, and good suites use all three.

Code graders are fast, cheap and reproducible. Did the test suite pass, does the record exist, is the JSON valid, was the refund under the limit. Their weakness is brittleness to valid variations.
Model graders use an LLM with a rubric to judge nuance, tone or completeness. They scale well but are not deterministic and must be calibrated. OpenAI warns that model judges can show position and verbosity bias, and recommends calibrating them against human labels.
Human graders are the gold standard and slow. Use them to build the labels that calibrate your model graders, and to review samples on a schedule.

Two grading habits save a lot of pain. First, grade outcomes, not paths. The Anthropic guide notes it is often better to grade what the agent produced than the exact steps it took, since rigid step checks punish valid approaches. Second, make graders hard to game, so passing requires actually solving the problem.

What is the difference between capability and regression evals?

They answer different questions and should be tracked separately.

Capability evals ask what the agent can do well. They should start with a low pass rate and give you room to climb.
Regression evals ask whether the agent still handles what it used to. They should sit near 100% and fail loudly when something breaks.

When a capability task becomes reliably solved, it graduates into the regression suite. When a capability suite saturates near 100%, it stops telling you anything about improvement, and it is time to add harder tasks.

How do evals fit into day to day work?

OpenAI puts it simply: evaluation is a continuous process. In practice that means:

Run the suite on every change to prompts, tools, models or reasoning settings, the same way you run unit tests.
Isolate every trial. Anthropic stresses that each trial should start from a clean environment, because shared state causes correlated failures and inflated scores.
Read transcripts. You will not know whether your graders are working unless you read the transcripts and grades from many trials. This is where most of the real insight comes from.
Record cost and latency per task. The same suite that measures quality tells you whether a cheaper model or a shorter prompt is safe. Without it, cost cutting is a gamble.
Feed production back in. Every confirmed failure in the wild becomes a new task.

Anthropic's guidance on building effective agents recommends extensive testing in sandboxed environments along with appropriate guardrails. Evals are how you turn that advice into something you can run every day. They also inform how much autonomy to give. This is one reason Coderhelm, our AI software engineer, turns GitHub issues and Jira tickets into draft pull requests: the work arrives as a proposal a person reviews, not a change that ships on its own.

An evals first checklist

Define success for the agent in one or two measurable sentences.
Collect 20 to 50 real tasks, including cases where the agent should refuse or do nothing.
Write a reference outcome for each task.
Add code graders for anything that can be checked mechanically.
Add a model grader with a written rubric, and calibrate it against a set of human labels.
Run several trials per task and report both pass@k and pass^k.
Start every trial from a clean, isolated environment.
Split the suite into capability and regression sets.
Track cost and latency per task alongside quality.
Read a sample of transcripts after every significant change.
Turn every production failure into a new task.

If you have an agent that seems to work but you cannot prove it, or you want to build one the measured way from the start, we would be glad to help. Start a project and tell us what the agent is supposed to get right.