Going deeper

Start by reading traces

Metrics chosen in advance measure what you imagined could go wrong. Error analysis starts from reality: read 50 to 100 real traces, write a note on the first thing that went wrong in each, then group the notes into categories and count them. The biggest category tells you what to fix, and which evaluation is worth writing. Hamel Husain and Shreya Shankar teach this approach, and it’s where most teams should start.

Pass or fail beats a score out of 5

A score from 1 to 5 invites debates about the difference between a 3 and a 4. A binary judgment forces you to state what a correct result is. Pair it with a sentence that explains why. Those sentences later become the examples for an LLM judge.

An LLM judge is a model to validate

Treat the judge as a classifier. Label examples by hand, keep some aside, and measure two rates: the true positive rate (it accepts what’s good) and the true negative rate (it rejects what’s bad). Raw agreement hides a lazy judge. On a 90% good sample, always saying “pass” agrees 90% of the time and catches no failures.

How many cases?

A pass rate needs an interval. The Wilson interval (1927) behaves well on small samples: 14 passes out of 15 reads as 93%, with a 95% interval of about 70% to 99%. So a change from 12/15 to 13/15 proves nothing. Agents also vary between runs, so repeat each case and report the unstable ones separately.

Check the path, not only the answer

An agent can give the right answer after doing something forbidden. Trajectory assertions check what it did:

const calls = trace.toolCalls.map((c) => c.name);
ok(calls.includes('search_policy'), 'policy checked');
ok(calls.indexOf('search_policy') < calls.indexOf('refund'), 'checked before paying');
eq(calls.filter((n) => n === 'refund').length, 1, 'paid once');

Prefer checks on what must be true, such as “checked before paying”, over an exact sequence of tools. A strict sequence fails each time the agent finds an equally valid path.

Evaluations as a gate

Add a case to the set for each production failure, so that a bug can’t return silently. Run a fast subset on each change, and the full set before each release. Block a release when a critical case fails, and track the overall score as a trend. Keep a held-out set you don’t tune against, because a set you’ve optimized for stops measuring anything.