The brief, and the first mistake
By the end: Identify, from a realistic brief, the point at which a team over-reached for autonomy.
Meridian Tools sells hand tools to trade customers. Four people handle support, about 400 tickets a week. Roughly half are the same three questions: where is my order, why was I charged twice, does this fit that. The brief from the founder was one line: "Make support faster without making customers angry."
The team built an agent. One system prompt, eleven tools covering orders, shipments, invoices, refunds, the docs and the ticket system, and an instruction to resolve the ticket end to end. It demoed beautifully. On the twelve tickets they tried by hand, it resolved eleven.
- 38% of tickets resolved without a human editing the reply — against 92% in the demo.
- Median 9 steps per ticket, and a long tail: one ticket ran 31 steps before hitting the cap.
- $0.47 per ticket. At 400 tickets a week that is about $9,800 a year for support that still needed checking.
- Two refunds issued that should not have been. Both recoverable, both embarrassing.
- One customer told, confidently, that an order had shipped when it had not.
The demo was not a lie; it was an unrepresentative sample. The twelve hand-picked tickets were well-formed questions from customers who gave their order number. Real tickets arrive as "hi, this is wrong again" with a photograph attached.
Note what was not wrong. The model was capable. The tools worked. The prompt was reasonable. The error was architectural, made before any code was written, and no amount of prompt tuning would have found it.