A voice agent that takes real phone calls needs a test suite that can call it, and building that suite taught us more about the limits of synthetic testing than about the agent itself.
Why you cannot just write assertions on the transcript
A voice conversation branches. The same intent — "cancel my booking" — arrives phrased a dozen different ways, interrupted, half-finished, or buried in a longer sentence about something else entirely. A test that checks for an exact transcript match is brittle within a week. We assert on outcome instead: did the booking get cancelled, was the caller told the correct status, did the call end in a sensible state.
The harness
- A library of real, anonymised call recordings — the actual variation callers produce, not what we imagine they would say.
- A synthetic caller — a second model playing a caller persona against the agent, useful for coverage but not for confidence on its own.
- Outcome assertions — checked against the backend state after the call, not against the words used to get there.
Where synthetic callers mislead you
A model playing a caller is a well-behaved caller. It does not get impatient, it does not talk over the agent, it does not have a bad line with dropped words, and it never has a background noise problem. Every one of those is common on a real support line and every one of those is where the agent's actual weaknesses show up. We treat synthetic-caller test runs as a floor, not a ceiling — passing them is necessary and nowhere near sufficient.
A test caller that never gets frustrated will never find the path where a frustrated human hangs up.
The failure mode that mattered most
The agent handled clear requests well and handled ambiguous ones by guessing rather than asking a clarifying question, which is the wrong default for anything touching a booking or a payment. We changed the fallback behaviour so that below a confidence threshold the agent asks once, plainly, rather than proceeding on its best guess — a small change with an outsized effect on the false-cancellation rate.
What we monitor once it is live
Real call recordings get sampled weekly and reviewed by a person, specifically for the cases the test suite could not have anticipated. The suite catches regressions; it does not discover new categories of failure. Only real callers do that, which is the whole argument for keeping a human in the loop after launch, not just before it.