Module 7 · Knowing It Works · lesson 31 of 37 · 8 min

Using a model as a judge

By the end: Use model-graded evaluation where appropriate and name the failure modes it brings.

Some outputs have no programmatic check. Was the reply appropriate in tone? Did the summary keep what mattered? For these, a model can grade — carefully, and knowing what goes wrong.

Give the judge a rubric rather than a question. Not "is this good" but "does it answer the question asked, does every factual claim appear in the source, is the tone appropriate for a customer, is it under 200 words" — each judged separately with a reason. Specific criteria produce judgements you can act on and disagree with.

So validate the judge itself. Grade fifty cases by hand, compare, and measure the agreement. If the judge and the humans agree eighty-five per cent of the time, that number is the ceiling on everything it tells you afterwards — and it is worth knowing before you make decisions on its output.

The strongest source of judgement is usually free and already in your system: the person who accepted, edited or rejected the agent’s work. Those decisions are real human labels on real cases, produced by people doing their job rather than a labelling exercise.

← Building AI Agents

Module 7 · Knowing It Works · lesson 31 of 37 · 8 min

Using a model as a judge

By the end: Use model-graded evaluation where appropriate and name the failure modes it brings.

Some outputs have no programmatic check. Was the reply appropriate in tone? Did the summary keep what mattered? For these, a model can grade — carefully, and knowing what goes wrong.