The ticket said the parcel arrived damaged, with a photo attached. The triage system correctly classified it as "damaged goods", correctly routed it to the returns team, and correctly attached the return policy. Nobody escalated it, and the customer waited four days for a resolution that should have taken four minutes.
Routing was never the problem
We built the triage layer to answer one question: which queue does this ticket belong in. It does that well — accuracy has been consistently high since launch. What it does not do, because nobody asked it to, is answer the second question: does this particular ticket need to skip the queue entirely.
The missing category
Some tickets are correctly classified and still urgent enough that the standard process is the wrong process. A photo of a crushed parcel with a customer who has ordered six times before is not "process this return in the usual three days" — it is "refund now and ask questions never", because the cost of the goodwill gesture is smaller than the cost of a good customer's third bad experience.
What we added
- A separate severity signal, independent of category — order history, sentiment in the message, and whether this is a repeat contact on the same order.
- A short list of auto-resolvable cases where severity is high and confidence is high: refund issued immediately, ticket closed, human notified afterwards rather than beforehand.
- Everything else still goes to a queue, because most tickets genuinely do not need speed, they need accuracy, and the two goals pull in different directions.
Why we did not just tell the model to "use judgement"
We tried that first. It produced inconsistent results because "judgement" was not actually defined anywhere — the model was guessing at a policy that existed only informally, in the heads of the two senior support agents. We wrote the policy down instead, as explicit rules with named thresholds, and the model applies those rules rather than improvising.
Classification tells you what something is. It does not tell you what to do about it, and treating the two as one step hides the decision nobody made.
Since the change
Auto-resolved refunds run at a small but steady rate, and the support team's own satisfaction with the tool went up more than the customer-facing metrics did — mostly because the obvious cases stopped clogging the queue that the hard cases still need.