A client asked us to review a document-summarisation feature built by a previous team, after users started noticing summaries that confidently described things the source document did not say. Nothing about the fix was exotic, which is worth saying plainly.
What we found first
The feature had no grounding check. The model was asked to summarise a document and given free rein over the wording, with no mechanism verifying that any claim in the summary could be traced back to the source text. It behaved exactly as a model does under those conditions: fluent, confident, and occasionally wrong in a way that read as authoritative.
The fixes were unglamorous
- Extractive anchoring — the summary now has to reference specific passages it draws from, and those passages are checked to exist in the source document before the summary is shown.
- A confidence-adjacent flag, not from the model itself but from the grounding check: any summary sentence that could not be matched to source text is marked for review rather than shown as fact.
- A test suite built from real, previously-flagged bad summaries, run against every change to the summarisation prompt going forward, so a fix could not silently regress.
What was not the problem
The model itself. We did not need a larger or more capable model to fix this — the underlying model, given a grounding constraint it did not previously have, was accurate on the same documents that had produced bad summaries before. The gap was architectural, not a matter of model quality.
Most of the AI feature failures we get asked to fix are not model failures. They are the absence of a check that would have been unremarkable to build if anyone had scoped it as part of the feature.
Why nobody caught it before shipping
The original team had tested the feature against documents chosen because they summarised cleanly. The bad cases only appeared with dense, ambiguous or contradictory source material — exactly the documents a busy user is most likely to want summarised, and exactly the ones most absent from a quick pre-launch test pass.
What we told the client to watch for next time
Any feature that turns unstructured model output into something a user reads as fact needs a grounding or validation step considered from the start, budgeted as part of the feature rather than added afterwards as a patch. It is far cheaper to build in the original sprint than to retrofit once users have already seen the wrong version.