The bug report was simple: a customer was billed for a car that was already returned. The cause took longer to find than the fix, and it was the same cause behind three other tickets we'd closed as one-offs.
The device reports intent, not fact
The lock firmware acknowledges the command it was sent — "lock requested" — well before it confirms the bolt has actually engaged. Our backend was treating the acknowledgement as the fact. Most of the time these are the same event a fraction of a second apart, which is exactly what makes the gap easy to miss in testing.
Where the gap shows up
- Poor signal. A confirmation that would normally arrive in under a second can arrive thirty seconds later, or not at all, and the backend needs an honest state for "unknown" rather than assuming the last command succeeded.
- Firmware updates change behaviour silently. A vendor update once changed the order in which two status fields were sent, and nothing in our contract with them caught it because there was no contract, just an assumption.
- Physical failure looks identical to a lost message. A jammed bolt and a dropped packet produce the same symptom from the backend's point of view: no confirmation.
What we changed
We separated "command sent", "command acknowledged" and "state confirmed" into three distinct fields with three distinct timestamps, and billing now keys off the confirmed state, with an explicit unresolved status when confirmation hasn't arrived within a threshold. An unresolved car is flagged to a human rather than silently treated as returned or not returned.
Treat every field coming off a physical device as a claim, not a fact, until something independent confirms it.
The customer-facing consequence
Unresolved states now surface as a small delay message rather than an incorrect charge. Customers tolerate "we're confirming this" far better than a refund request, and the number of billing disputes tied to this class of bug dropped to zero once the unresolved state existed as a first-class thing rather than a bug.
The broader lesson for hardware-backed products
Any integration with a physical device benefits from the same three-state discipline: requested, acknowledged, confirmed. It costs a bit more schema and a bit more UI, and it removes an entire category of "the app says one thing and the world says another" complaints.