Clicking through an interface to check whether a model-backed feature behaves correctly is slow, and it conflates two separate questions: does the reasoning work, and does the interface present it correctly. We now separate them from the start.
Test the contract, not the chat
Before any screen is built, we write tests directly against the function that calls the model — fixed inputs, expected shape of output, and specific assertions about content where the domain allows it. This runs in seconds and does not require a browser, a UI framework, or a person clicking anything.
What we assert against
- Schema conformance — the output validates against the type it is supposed to produce, every time, including on adversarial input designed to break it.
- Grounding — where the feature is meant to answer from specific data, we assert the answer actually cites or matches that data, not just that it sounds plausible.
- Refusal and fallback behaviour — what happens when the model cannot answer confidently, tested as deliberately as the happy path.
- Stability across repeated runs — the same input run several times should produce outputs that agree on anything that matters to the business, even if the wording varies.
Where non-determinism forces a different kind of test
A model does not return the same string twice, so exact-match assertions are the wrong tool. We assert on structure and on specific required facts being present, and we accept variation in phrasing as out of scope for testing — that variation is a product and tone question, not a correctness question, and belongs to a different kind of review.
Testing a model-backed feature by clicking through a UI answers "did this example work". Testing the contract directly answers "does this hold in general". Only one of those catches the next regression.
What this caught before a UI even existed
On one project, contract-level tests found that a change to the prompt caused the model to occasionally omit a required field under a specific combination of inputs — reproducible perhaps one run in twenty. That is exactly the kind of intermittent failure that a person clicking through a demo would very likely miss and a script running the same input five hundred times would not.
Where UI testing still matters
Once the contract is solid, we do test the interface — loading states, error presentation, what happens when a response arrives slowly. That is a different, smaller set of tests, and it is far easier to write once you are confident the underlying reasoning is not also broken.