The request was "an AI assistant for support". The team was drowning in a queue with a wide spread of issue types and a small number of genuinely hard cases. A chat box would have added a surface, not removed work.
What we measured first
Before designing anything: how many tickets per day, how long each takes, and what the time is actually spent on. The answer was that roughly 60% of handling time went into reading, categorising and finding the right precedent — not into writing the reply.
That number determined the design.
Three pieces
- Classification into the existing taxonomy the team already used, with a confidence score. No new categories invented by a model.
- Retrieval of the three most similar resolved tickets, shown alongside the new one with links.
- A drafted reply, always editable, never sent automatically, with the source tickets cited underneath.
What we deliberately did not build
Auto-send. Sentiment scoring nobody would act on. A customer-facing bot. Each was discussed and each failed the same test: what happens when it is wrong, and who notices.
The model should shrink a task, not add a new surface to learn.
Cost and correction
Cost per ticket was capped in design and tracked from the first sprint. Correction rate — how much of the draft agents rewrote — became the quality metric, reviewed monthly. Where it stayed high for a category, we removed drafting for that category rather than tuning forever.
Result
Median handling time fell by roughly a third within two months, concentrated exactly where the measurement said it would be: the reading and precedent-finding step. Reply quality, judged by the team's existing review sample, did not move.