Module 9 of 10, lesson 3 of 5

Videos

Evaluating an agent

Watch these first

How to evaluate agents in practice

A three-tier pyramid: tool and parameter tests, trajectory, then human review. Google Cloud Tech, 10:53

How to evaluate agents in practice Play video (10:53, Google Cloud Tech)

Clicking loads the YouTube player, which may set trackers.

• 1:05 The testing pyramid
• 1:55 Trajectory tests
• 4:48 Demo of all three levels

AI Evaluations Clearly Explained in 50 Minutes (Real Example)

The work most teams skip: reading traces in a spreadsheet, taking notes and grouping them before writing any metric. Peter Yang, with Hamel Husain, 52:30

AI Evaluations Clearly Explained in 50 Minutes (Real Example) Play video (52:30, Peter Yang, with Hamel Husain)

Clicking loads the YouTube player, which may set trackers.

• 1:25 Analyzing 100 real traces
• 9:50 Writing criteria in a spreadsheet
• 24:44 Why pass/fail beats scores
• 28:52 The agreement metric trap

LLM as a Judge: Scaling AI Evaluation Strategies

Rubric or pairwise comparison, and the biases to test: position, length and self-preference. IBM Technology, 6:09

LLM as a Judge: Scaling AI Evaluation Strategies Play video (6:09, IBM Technology)

Clicking loads the YouTube player, which may set trackers.

• 3:45 The judge’s biases
• 5:15 Detecting those biases

If you want more

Optional, not counted in the module time.

How to Systematically Setup LLM Evals

A full evaluation method on a ticket-handling project: unit tests, human review, then judges aligned on humans. Dave Ebbelaar, 55:02

How to Systematically Setup LLM Evals Play video (55:02, Dave Ebbelaar)

Clicking loads the YouTube player, which may set trackers.

• 12:26 Levels of evaluation
• 14:01 Unit tests for LLMs
• 22:44 Aligning LLM evaluators
• 37:25 Common mistakes

Legal notice and privacy Manage cookies