A demo of an AI feature shows the output. A test suite for one has to cover everything that produced it, and most of that machinery is invisible to anyone clicking through the product.
The parts we now test explicitly
- Prompt assembly — given a fixed set of inputs, does the assembled prompt actually contain what we think it contains. This sounds trivial and catches real bugs: a template variable silently rendering empty, a truncation limit cutting off the part of the context that mattered.
- The retry and fallback path — what happens on a timeout, a malformed response, a rate limit. This path runs rarely in production and therefore gets almost no organic testing unless someone writes it deliberately.
- Logging fidelity — when something goes wrong three weeks from now, will the log contain enough to reconstruct what the model actually saw. We have debugged incidents that took days longer than they should have because the log had the output but not the input.
Why the visible output is the least useful thing to test heavily
Model output for a given input can vary run to run even with the same weights and settings, depending on the provider. Asserting on exact output text is asserting on the wrong layer. We assert on the schema the output must satisfy and on the downstream behaviour it triggers, which is stable even when the wording is not.
A short example
A support-reply feature we built had a suite that was almost entirely output-text assertions and passed reliably in CI while failing intermittently for users. The actual bug was in the retry path — a failed call was retried with a stale copy of the conversation context, so the second attempt answered a different, earlier question. No output-text test would ever have found that, because it was testing the wrong layer entirely.
The visible output of a model-backed feature is a sample from a distribution. Testing the plumbing around it is testing something that actually holds still.
What this changes about test writing
Someone on the team who has never called a model API can still write and maintain most of this suite, because it is testing conventional code — templating, control flow, error handling. That was not obvious to us at the start, and it is the reason the AI features on our recent projects have test coverage comparable to the rest of the codebase rather than a thin, output-only layer bolted on separately.