A traditional test says: given this input, expect exactly this output. A model-backed feature makes that assertion mostly useless, because a reasonable model can phrase the same correct answer three different ways in three runs. We had to work out what to check instead.
Separate the parts that should be deterministic
The first move is architectural, not testing-related: as much of the feature as possible should not go through the model at all. Retrieval, validation, formatting, business rules — all of that stays in ordinary code with ordinary tests. The model's job shrinks to the genuinely open-ended part, which shrinks the surface area we need this different kind of test for.
What we assert about the model's output
- Structure, always. If the contract says a JSON object with three named fields, the test asserts the shape, not the prose inside it.
- Constraints, explicitly. Does the output stay within a length limit, avoid a list of forbidden claims, cite only sources it was actually given.
- Properties, not values. For a summariser: is every named entity in the output also present in the input. That catches fabrication without requiring an exact match to a reference summary.
- Known failure cases, as regression tests. Every time a real output was wrong in a specific way, that input becomes a permanent test case, checked for the specific failure recurring rather than for an exact match.
Where a second model earns its keep
For softer qualities — tone, coherence, whether an answer actually addresses the question — we use a second model as a grader, with a narrow rubric and a numeric threshold, run against a fixed set of inputs on a schedule rather than on every commit. It is not as trustworthy as a deterministic assertion, so we do not treat it as a merge gate. It is a trend line, and a trend moving the wrong way gets a human's attention.
You cannot assert the exact words a model will produce, and trying to is how teams end up with brittle tests that fail on a harmless rephrasing and pass on a genuine regression.
The human sample that never goes away
A fixed number of live outputs get read by a person every week, regardless of what the automated checks say, because some failures are boring to a rubric and obvious to a human — an answer that is structurally valid, passes every property check, and is nonetheless slightly patronising in tone. No test we have written catches that reliably yet.
What this bought us
Confidence to change the underlying model or prompt without re-litigating every case by hand, because the property tests and the regression set catch the failures that matter, and the weekly read catches the ones that don't fit a rule yet.