Most of the interesting engineering on a voice dispatch project turned out to be adjacent to the model, not inside it. Here is the part of the build that never appears in a demo.
The model was the easy part
Given a transcript and a set of intents — book a ride, cancel, ask where the car is — a well-grounded model handles the language reliably. The harder problems were about the call itself, not the conversation inside it.
Latency has a personality
A pause of six hundred milliseconds reads as thoughtful in a text interface. On a phone it reads as a dropped call, and the caller starts talking again, which produces a transcript with two people's speech overlapping and a model that now has to guess which half to answer. We spent more time on interruption handling — detecting that the caller has started speaking and yielding immediately — than on anything to do with intent recognition.
Silence is a valid response
The instinct when building a voice agent is to fill every gap. We removed several confirmations the agent used to say out loud, because "booking your ride now" while the booking API is already returning made the agent sound like it was narrating its own competence rather than doing the job.
The failure path, again
- Ambiguous address — the agent asks one clarifying question, not three.
- API timeout — the caller hears a genuine "one moment", not a synthetic filler loop, and a human is paged if it exceeds a few seconds.
- Caller clearly frustrated — detected by repetition and raised volume, handed to a person without asking permission first.
The measure of a voice agent is not how natural it sounds when everything works. It is how quickly it gets out of the way when something doesn't.
What surprised the client
Not the automation rate, which they had budgeted for. The thing that changed their mind about the project was the call recordings from weeks two and three, where the agent handed off correctly to a human dispatcher on calls it had no business finishing — a caller asking about a lost item, a driver complaint. Getting the hand-off right earned more trust than getting the booking flow right.