We started attaching a per-operation cost to every feature that calls a model, and promised a review once there was a month of data. The review happened last week, on a call with three engineers and a spreadsheet nobody enjoyed opening.
The number that moved
One feature — a document summariser inside a client's back office — was costing four times its estimate. Not because the model got more expensive, but because the average input had grown: users had started pasting whole email threads instead of single messages, and nobody had put a limit on that.
Estimates assume a shape of input that drifts
The estimate was built from a sample of real documents collected during discovery. Reasonable at the time. What we missed is that usage patterns are not static — once a feature works, people feed it more than the sample suggested, because now it's useful enough to bother.
- Cap the input where the product allows it, and say so in the interface rather than truncating silently.
- Chunk and summarise progressively for the cases where the input genuinely needs to be large.
- Re-sample monthly, not once at discovery, because the shape of usage changes as the feature earns trust.
The features that stayed on budget
The two features that tracked their estimate closely both had a fixed-shape input: a form with defined fields, not free text. That is not a coincidence. Cost predictability tracks input predictability far more than it tracks the size of the model.
A cost estimate made during discovery is a hypothesis about how people will use the feature, not a fact about the model.
What we changed for February
Every model-backed feature now gets a cost dashboard before it ships, not after a client asks why the bill went up. It is three numbers: average cost per call, call volume, and the 95th percentile cost, because averages hide the one user who pastes in a novel.
The uncomfortable part
Two of these reviews end with the same conclusion: the feature is good, and it is too expensive for how it is currently used. That is a product conversation, not an engineering one, and it goes back to the client rather than getting quietly absorbed into a hosting bill. Nobody wants to have that conversation, but it beats finding out from an invoice.