Looking back across six AI features scoped and shipped this year, the questions that actually predicted whether the feature went well were rarely about the model at all.
The question that mattered most: what happens when it's wrong
Every one of the six features needed a real answer, agreed before build, to "what does the user see when the model gets this wrong". The two features where that answer was vague going into build were the two that needed the most rework afterwards — not because the model performed worse, but because nobody had designed for its failure mode until users found it for them.
The second question: who is accountable for the output
Model output that reaches a user needs an owner in the same way a paragraph of copy written by a person does. On one project this was explicit from day one — a named team reviewed sample outputs weekly. On another it was nobody's job, by omission rather than decision, and quality drifted for a month before anyone noticed.
What actually varied by feature, and it wasn't accuracy
- Latency tolerance — a background summarisation job can take ten seconds; an inline suggestion while typing cannot take more than a few hundred milliseconds, and that constraint shaped the architecture far more than anything about the model's capability.
- Reversibility — a suggested email draft is trivially reversible if wrong; an automated stock reorder is not, and the two demanded completely different levels of human confirmation before acting.
- How the user would notice a wrong answer — some outputs are checkable at a glance against something the user already knows; others are opaque, and opaque outputs need a much higher bar before shipping without review.
The pattern across all six
None of the features that went well did so because the underlying model was unusually capable for the task. They went well because the failure mode, the accountability, and the reversibility were designed deliberately rather than discovered in production.
The model is the easiest part of an AI feature to get right and the part everyone spends the most time discussing. The scoping questions that actually determine the outcome are the ones about what happens around it.
What our scoping document looks like now
Every AI feature ticket now has a section, alongside the usual acceptance criteria, that answers these questions explicitly before implementation starts. It has added perhaps an hour to the scoping of each feature. Measured against the rework the two vague ones needed, that hour is the cheapest part of the whole process.