Module 9 of 10, lesson 4 of 5
Practice
Evaluating an agent
Write the code
Runs in your browser, with no install, account or API key. The model is scripted, so the results are repeatable.
function wilson(passes, n, z = 1.96) { // TODO return { low: 0, high: 0 };}function clearlyBetter(a, b) { // TODO return false;}Give the agent a skillPut an interval on a score
Open in the editorasync function evaluate(cases, agent, { runs = 3 } = {}) { let passes = 0; const failures = []; for (const testCase of cases) { for (let run = 0; run < runs; run += 1) { try { const output = await agent(testCase.input); if (testCase.check(output)) passes += 1;Repair the agentFix a dishonest eval runner
Open in the editorfunction calibrate(labels, threshold = 0.9) { // TODO return { agreement: 0, tpr: null, tnr: null, trustworthy: false };}module.exports = { calibrate };Give the agent a skillCalibrate an LLM judge
Open in the editorRunning project
Running project: write 20 cases for the refund agent, 5 of them traps. Replay them 3 times. Record the pass rate, the average cost per case and the causes of failure. Fix one cause and rerun: did the score move more than the noise?
Practice
- Evaluating AI Agents DeepLearning.AI
Free enrollment: 15 video lessons, 6 code examples, 1 graded assignment.
Some of these need a free account or an API key, which may be paid.