Skip to content
← Writing
Writing · October 2026 · 2 min read

Your model changed and nobody told you

Models can shift without notice, and a new version can change how your prompts are handled. Keep a test set so you are surprised before your users are.

Abhijit Chatterjee Chattopadhyay

The Logic Atelier logo above the words: Same name. New answers. Your model changed and nobody told you.

Your AI feature can start behaving differently without a single line of your code changing.

Sam Newman again, from the same chapter of Building Resilient Distributed Systems I quoted yesterday. Yesterday was the provider going down. Today is the quieter failure: the provider stays up, and the answers change.

Same name, different behaviour

In the excerpt published by The Pragmatic Engineer (opens in a new tab), Newman writes that "small changes in models can also lead to quite different behavior." He has seen reports of a model starting to behave differently with no major change flagged by the vendor. And moving to a newer version "can fundamentally change how your prompts may be handled."

Nothing errors. Your dashboards stay green. The answers are just a little different, and you hear about it from a customer.

Prompts do not travel

The same applies when you switch on purpose. Newman warns that "a prompt tuned against one model will have to be reevaluated when moving to a different model, even if you stay within the same vendor ecosystem."

So the second route from yesterday's note is only a real fallback once you have tested your prompts on it.

Be surprised first

His answer is tests: traditional functional tests together with AI evaluations, usually called evals, which score the model's answers against what you expect. The aim, as he puts it, is that when the unexpected happens, you are the one surprised first, not your users.

In practice that means three things:

  1. A fixed test set. Real inputs with the answers you would accept, including the awkward ones.
  2. Checks a machine can run. Format, required fields and banned content by rule, and a scored comparison for the rest.
  3. A schedule, not a ceremony. Run it regularly against the live model, and before any model or prompt change.

When the scores move and you changed nothing, the model did.

Do this today

  • List which model each AI feature calls, and whether that name can point to a new version without you choosing it.
  • Pin a specific version where your provider offers one.
  • Save 50 real inputs and the answers you would accept.
  • Run them once now and keep the result as your baseline.

Your next step

If you want model changes caught before your users see them, book a free 30-minute 1:1 assessment. I look at your setup with you and tell you what the work would take.

Drafted with AI help, edited and approved by me.

New articles by email

I will never sell your details, for any reason, and I never send spam.

How I use your details
  • I, Abhijit Chatterjee Chattopadhyay of The Logic Atelier, hold them, in my own newsletter system hosted on Fly.io. Emails go out through Resend.
  • Your name is for greeting you, your email is for sending new articles, and your phone number, if you give it, is so I can reach you directly about my work.
  • If you do not confirm your email within 7 days, I delete the sign-up. If you unsubscribe, I delete your details 30 days later.
  • If Google Analytics is running on your visit, it is told that a sign-up was made, and nothing you typed.
  • To see or delete what I hold about you at any time, email hello@thelogicatelier.com.
  • The full detail is on the privacy page.
NextYour AI provider will go down. Then what?
← cd ../writing

Your model changed and nobody told you

model-changed.mdfront matter
title: Your model changed and nobody told you
date: 2026-10-23
reading_time: 2 min
author: Abhijit Chatterjee Chattopadhyay
model_changed_cover_52742fa6_cb7ebbc965.pngcover
The Logic Atelier logo above the words: Same name. New answers. Your model changed and nobody told you.

Your AI feature can start behaving differently without a single line of your code changing.

Sam Newman again, from the same chapter of Building Resilient Distributed Systems I quoted yesterday. Yesterday was the provider going down. Today is the quieter failure: the provider stays up, and the answers change.

Same name, different behaviour

In the excerpt published by The Pragmatic Engineer (opens in a new tab), Newman writes that "small changes in models can also lead to quite different behavior." He has seen reports of a model starting to behave differently with no major change flagged by the vendor. And moving to a newer version "can fundamentally change how your prompts may be handled."

Nothing errors. Your dashboards stay green. The answers are just a little different, and you hear about it from a customer.

Prompts do not travel

The same applies when you switch on purpose. Newman warns that "a prompt tuned against one model will have to be reevaluated when moving to a different model, even if you stay within the same vendor ecosystem."

So the second route from yesterday's note is only a real fallback once you have tested your prompts on it.

Be surprised first

His answer is tests: traditional functional tests together with AI evaluations, usually called evals, which score the model's answers against what you expect. The aim, as he puts it, is that when the unexpected happens, you are the one surprised first, not your users.

In practice that means three things:

  1. A fixed test set. Real inputs with the answers you would accept, including the awkward ones.
  2. Checks a machine can run. Format, required fields and banned content by rule, and a scored comparison for the rest.
  3. A schedule, not a ceremony. Run it regularly against the live model, and before any model or prompt change.

When the scores move and you changed nothing, the model did.

Do this today

  • List which model each AI feature calls, and whether that name can point to a new version without you choosing it.
  • Pin a specific version where your provider offers one.
  • Save 50 real inputs and the answers you would accept.
  • Run them once now and keep the result as your baseline.

Your next step

If you want model changes caught before your users see them, book a free 30-minute 1:1 assessment. I look at your setup with you and tell you what the work would take.

Drafted with AI help, edited and approved by me.

subscribeemail

// I will never sell your details, for any reason, and I never send spam.

How I use your details
  • I, Abhijit Chatterjee Chattopadhyay of The Logic Atelier, hold them, in my own newsletter system hosted on Fly.io. Emails go out through Resend.
  • Your name is for greeting you, your email is for sending new articles, and your phone number, if you give it, is so I can reach you directly about my work.
  • If you do not confirm your email within 7 days, I delete the sign-up. If you unsubscribe, I delete your details 30 days later.
  • If Google Analytics is running on your visit, it is told that a sign-up was made, and nothing you typed.
  • To see or delete what I hold about you at any time, email hello@thelogicatelier.com.
  • The full detail is on the privacy page.

vi.Contact

06 · contact

Talk to me.

Talk to me.

Production AI and software architecture. Based in Dubai, working with teams in the Emirates and Ireland.

I work Dubai hours, Monday to Friday, the same week you do. Dubai is three hours ahead of Dublin in summer and four in winter, so my afternoon is your morning.

I will never sell your details, for any reason, and I never send spam.

How I use your enquiry
  • Your enquiry comes to me, Abhijit Chatterjee Chattopadhyay of The Logic Atelier. The form is handled by Formspree, which passes it to my inbox and keeps a copy in my Formspree account.
  • I use it to read your enquiry and reply. If no work follows, I delete it within 12 months, and sooner whenever you ask.
  • If Google Analytics is running on your visit, it is told that an enquiry was sent, and nothing you typed.
  • To see or delete what I hold about you, email hello@thelogicatelier.com.
  • The full detail is on the privacy page.