Your model changed and nobody told you
Models can shift without notice, and a new version can change how your prompts are handled. Keep a test set so you are surprised before your users are.
Abhijit Chatterjee Chattopadhyay

Your AI feature can start behaving differently without a single line of your code changing.
Sam Newman again, from the same chapter of Building Resilient Distributed Systems I quoted yesterday. Yesterday was the provider going down. Today is the quieter failure: the provider stays up, and the answers change.
Same name, different behaviour
In the excerpt published by The Pragmatic Engineer (opens in a new tab), Newman writes that "small changes in models can also lead to quite different behavior." He has seen reports of a model starting to behave differently with no major change flagged by the vendor. And moving to a newer version "can fundamentally change how your prompts may be handled."
Nothing errors. Your dashboards stay green. The answers are just a little different, and you hear about it from a customer.
Prompts do not travel
The same applies when you switch on purpose. Newman warns that "a prompt tuned against one model will have to be reevaluated when moving to a different model, even if you stay within the same vendor ecosystem."
So the second route from yesterday's note is only a real fallback once you have tested your prompts on it.
Be surprised first
His answer is tests: traditional functional tests together with AI evaluations, usually called evals, which score the model's answers against what you expect. The aim, as he puts it, is that when the unexpected happens, you are the one surprised first, not your users.
In practice that means three things:
- A fixed test set. Real inputs with the answers you would accept, including the awkward ones.
- Checks a machine can run. Format, required fields and banned content by rule, and a scored comparison for the rest.
- A schedule, not a ceremony. Run it regularly against the live model, and before any model or prompt change.
When the scores move and you changed nothing, the model did.
Do this today
- List which model each AI feature calls, and whether that name can point to a new version without you choosing it.
- Pin a specific version where your provider offers one.
- Save 50 real inputs and the answers you would accept.
- Run them once now and keep the result as your baseline.
Your next step
If you want model changes caught before your users see them, book a free 30-minute 1:1 assessment. I look at your setup with you and tell you what the work would take.
Drafted with AI help, edited and approved by me.