Traditional acceptance criteria assume a deterministic system: given this input, this output. A model-backed feature breaks that assumption at the one place teams are least prepared for it to break — the acceptance test itself.
The temptation to skip criteria entirely
Faced with non-deterministic output, some teams give up on acceptance criteria for the AI parts of a feature and fall back to "looks reasonable in a demo", which is not a standard, it is the absence of one wearing a standard's clothes.
What actually is testable
- Structural correctness — does the output conform to the schema it must satisfy, every time, regardless of content.
- Behavioural bounds — does the system do the right thing with the output, for a representative range of plausible outputs, including malformed or low-confidence ones.
- Refusal and escalation criteria — under what conditions should the system decline to act automatically and hand off to a person, stated explicitly rather than left to whatever the model happens to do.
- A sampled quality bar — a fixed set of representative inputs, evaluated against a rubric by a person, run whenever the prompt or model changes, so quality regressions are caught even though no single output is "correct" in isolation.
Writing this into a ticket
A ticket for a model-backed feature now states its schema, its escalation conditions and its rubric before implementation starts, in the same acceptance section as every other ticket. The rubric is short — three or four questions a reviewer can answer about a sample output — and it is the artefact that turns "does this feel right" into something two different reviewers would score the same way.
You cannot write an acceptance test that pins down a specific sentence a model will produce. You can write one that pins down what must be true about any sentence it produces.
Where this caught a real problem
A drafting assistant we built passed every structural test and every escalation-condition test, and still shipped with outputs that were technically compliant and unhelpfully terse. The rubric review, run against a sample before launch, is what caught that — a category of defect neither the schema check nor the escalation logic was designed to see, because terseness is not a structural failure.
The general lesson
Probabilistic components need acceptance criteria with more parts, not none. The parts that were always deterministic — schema, escalation, bounds — get tested exactly as before. The part that genuinely varies gets a rubric and a person, on a schedule, rather than being quietly excused from having a standard at all.