What to measure
By the end: Choose an end-to-end success measure for an agent and avoid measuring the wrong thing.
The most common evaluation mistake is measuring the middle: whether the agent picked the tools you would have picked, whether its reasoning reads well. Those are proxies, and agents routinely reach the right outcome by an unexpected route.
Prefer checks a computer can run. A test suite, a schema validation, a comparison against a known-correct record, a check that every cited id exists. Programmatic checks are cheap, repeatable and not subject to anyone’s mood, and you can run them on every change instead of occasionally.
Alongside the outcome, track cost and steps per run and the rate at which a person accepts what the agent produced. An agent that succeeds ninety per cent of the time at four times the budget is a different product from one that succeeds eighty-five per cent of the time cheaply, and only the numbers make that choice discussable.