Building an eval set from real failures

The instinct is to write a hundred test cases before launch. The better move is to write a handful, ship carefully to a small audience, and let reality tell you what to test. Invented failures are rarely the ones that happen.

Start with twenty cases: a dozen ordinary, a few odd, a few that should be refused or escalated. Run them on every prompt change, every tool change and every model change. Twenty cases you actually run beat two hundred you ran once.

Keep cases as data — input, expected outcome, and how to check it — rather than as code, so adding one is a small act anybody on the team can do. And keep a few that must fail: an injection attempt that must be ignored, a request outside scope that must be refused. Systems drift toward helpfulness, and these catch the drift.

Agents are non-deterministic, so a single pass is weak evidence. Run each case a few times and look at how often it succeeds. A case that passes three times out of five is not passing; it is a coin flip you have not noticed yet.