The demo shows the model getting it right. The product has to work on the day it gets it wrong, in front of a user with a deadline.
Ask the question early
During scoping, for every model-backed feature: what happens when this is wrong? If the answer is "the user notices and fixes it", show your work. If the answer is "nobody notices", the feature needs a different design or should not exist.
Patterns that hold up
- Show the source. Every extracted value links back to where it came from — the region of the scan, the paragraph in the document. Verification becomes a glance instead of a re-read.
- Make editing the default. Output lands in a form, not in a database. The user confirms.
- Mark confidence honestly. Low-confidence fields are visibly unresolved rather than quietly wrong.
- Fail to the old way. When the model is unavailable, the manual path still exists. It is also the fallback during an incident.
A feature that saves ten minutes and creates an untraceable error has not saved anything.
The measurement
We instrument correction rate: how often a user changes what the model produced, and on which fields. That number tells you where to improve the prompt, where to tighten the schema, and when a field should stop being automated at all.
Silent automation is the risk
The dangerous features are the ones with no human in the loop and no audit trail. If nothing in the product can answer "why does this record say that", the feature is a liability regardless of its accuracy.