Asked to write a date parser for a client whose primary market uses day-month-year, a model kept defaulting to month-day-year in edge cases, no matter how the ticket phrased the requirement. The mistake wasn't random. It was a fair reflection of which format shows up more often in the examples a general-purpose model has seen.
Mistakes have a shape
Random errors are frustrating but not informative. The errors we actually see cluster: date formats defaulting to the more common convention in public code, currency handling assuming two decimal places when a client's system uses none, pluralisation assuming English rules in code that has to handle several languages. Each one is explainable once you notice the pattern, and each one is a clue about what to state explicitly rather than assume is obvious.
What this changes about how we write tickets
- State the locally unusual convention explicitly, every time, even if it feels repetitive — assuming it's "obvious from context" is exactly the assumption that fails.
- Add a test for the case a model is statistically likely to get wrong, not just the case a human is likely to get wrong; they are not always the same case.
- Treat a recurring category of mistake as a documentation gap, not a series of unrelated bugs — if the date format keeps needing correcting, it belongs in the conventions file, not in five separate ticket comments.
The upside of this framing
Once a mistake is understood as "the statistically likely default, stated as a requirement", it stops being frustrating and starts being useful information about where the codebase's conventions diverge from the general case — which is often useful to know for human contributors too, particularly new joiners making the same assumption for the same reason.
A model's confident wrong answer is not evidence of unreliability. It's evidence of what wasn't said clearly enough.
Where this doesn't apply
Not every mistake is a training-data artefact — plenty are ordinary bugs, missed edge cases, or the model simply not having enough context to know better. The distinction matters because the fix is different: a training-data-shaped mistake gets fixed by stating the convention; an ordinary bug gets fixed by giving more context or fixing the ticket.
What we did about the date parser
Added the locale requirement explicitly to the ticket, added a test with an ambiguous date that would fail loudly under the wrong assumption, and added the convention to the project's conventions file so it wouldn't need restating on the next ticket that touched dates.