The lock said the car was locked. The car was not locked. Nobody had done anything wrong — the device had sent its acknowledgement before the actuator finished moving, because that is what its firmware does, and no amount of arguing with the manufacturer changes it this quarter.
The assumption we had to drop
It is tempting to model a hardware event as a fact: the door is now locked, the ignition is now off, the vehicle is now here. In practice every one of those is a claim made by a device with its own retry logic, its own clock, and its own idea of "now", arriving over a connection that drops mid-city on a Tuesday for no discoverable reason.
Treating signals as evidence, not state
- Every signal gets a confidence window, not just a value — a lock event less than two seconds after the unlock command is treated differently from one thirty seconds later.
- State changes require corroboration where the stakes are high: a car is only "in use" once both the door and ignition events agree, because either alone has a known false-positive rate.
- Timestamps are the device's, not the server's, and we log both, because the gap between them is itself useful telemetry about a flaky connection.
Reconciliation is a background job, not an afterthought
We run a job every few minutes that compares the state we believe against the last few signals from each device, and flags disagreement for a human rather than silently picking one. This looks like unnecessary caution until the day a lock genuinely fails and the alternative is a customer standing next to an unlockable car with no idea why.
A sensor tells you what it thinks happened. Deciding what actually happened is your code's job, not the sensor's.
What this cost us
Roughly two extra weeks of engineering time that a naive integration would not have budgeted for, spent entirely on the reconciliation layer and its tests. It has not been the source of a single production incident since, which is a lower bar to clear than it sounds — the naive version generated one every few weeks.
The broader pattern
Every hardware integration we have built — meters, roof lights, card terminals, door locks — eventually needed this same layer. We now scope it in by default rather than waiting to discover we need it, because we always do.