A provider rolled a model update behind an endpoint we'd been calling for months. No error rate changed, no latency changed, and for three weeks nobody noticed that one specific classification task had quietly got worse, because nothing was watching its accuracy directly.
What we were actually monitoring
Uptime, latency, and cost — the metrics that are easy to instrument and that most teams reach for first. None of them moved. The thing that moved was correctness on a specific task, which we had never put a number on because it had always just seemed to work.
Why "it always just seemed to work" is a dangerous state
A model-backed feature can degrade in accuracy without tripping any conventional alert, because accuracy isn't latency, isn't an error code, and isn't cost. It requires a ground truth to compare against, which most teams don't build until after the first time they get burned by not having one — which is roughly what happened here.
What we added
- A small, fixed evaluation set per model-backed feature, run against every model response as a periodic spot check, with a known correct answer for each item.
- An accuracy trend line on the same dashboard as cost and latency, so a drop is visible in the same glance rather than requiring someone to remember to go looking.
- A changelog subscription for every provider we depend on, however unglamorous, because a silent model update is silent to us only if we're not told when the provider makes one.
Why we didn't just pin the model version
Some providers allow pinning, some don't, and pinning trades one risk for another — a pinned model eventually gets deprecated on the provider's timeline rather than ours, often with less notice than a routine update. We pin where it's available and rely on the evaluation set everywhere, because the eval set is useful regardless of whether pinning is an option.
Cost and latency tell you the feature is running. They tell you nothing about whether it's right.
What the eval set actually caught
A roughly one-in-six drop in accuracy on the specific classification task, invisible in every other metric, traced to the update once we had a number to compare before and after. We reverted where pinning was possible and adjusted the prompt where it wasn't, recovering most of the gap.
What we'd tell a team building their first model-backed feature
Build the evaluation set before launch, not after the first time something quietly goes wrong. It's a small amount of upfront work that turns "it always just seemed to work" into something you can actually check.