Rebuilding around what is actually known
By the end: Restructure an over-autonomous system into a router with one genuinely agentic branch.
The rebuild started with a week of reading tickets rather than writing code. The team classified 300 of them by hand and found the distribution that decided the architecture.
- 41% — "where is my order": needs the order and the tracking record. Two lookups, always the same two.
- 18% — "does this fit / which one do I need": needs the product docs. One retrieval, then an answer.
- 14% — billing disputes: needs the order, the invoice, sometimes the payment record. Genuinely variable.
- 19% — everything else, including complaints and things that are not questions.
- 8% — not really support: partnership offers, job applications, spam.
That distribution is an architecture. The first 59% has a knowable shape, so it became two workflows: classify, then fetch exactly what that class needs, then draft. Fixed paths, two model calls each, with a gate that refuses to draft if the lookup came back empty.
The 14% of billing disputes stayed agentic, because the number of systems to consult genuinely depends on what the first one says. That branch kept a loop, a six-step cap and read-only tools.
The last 27% went straight to a person with a one-line summary attached. The team resisted this for a while — it felt like giving up — until someone pointed out that a fast, accurate handoff is a feature, and that "everything else" is where an agent is least likely to be right and most likely to be confidently wrong.