A judge that agrees with humans 90% of the time can still be useless: if 90% of outputs pass, always saying “pass” scores 90%. So measure both directions against human labels.

Implement calibrate(labels). Each item is { human, judge }, both booleans, where true means pass. Return:

agreement: share of items where they match,
tpr: among human passes, the share the judge passed,
tnr: among human failures, the share the judge failed,
trustworthy: true when both tpr and tnr reach threshold (second argument, default 0.9).

A rate with no items to measure is null, and then trustworthy is false.

Exercises in this module

Terminal