Skip to content
← Writing
Writing · October 2026 · 2 min read

The small model that beat the LLM

GitHub replaced a large language model with a small one. It raised alert acceptance 11.5% and cut serving cost over 98%. When to do the same.

Abhijit Chatterjee Chattopadhyay

The Logic Atelier logo above the words: The small model that beat the LLM.

GitHub took a large language model out of one of its security tools and put a much smaller one in. Developers accepted more of its alerts, and serving it costs over 98% less.

What GitHub changed

GitHub's secret scanning looks for passwords and keys left in code. The hard part is the lookalikes: a test value, a placeholder in a guide, a checksum. Flag those and developers stop trusting the alerts.

According to GitHub's write-up (opens in a new tab), it replaced its previous LLM-based pipeline with a small classifier fine-tuned from ModernBERT. In its customer test against the old pipeline:

  • 11.5% higher acceptance of its alerts.
  • 86% lower median scan time after code is pushed.
  • More than 98% lower serving cost.

Offline, on public traffic, it also reported 13% higher precision and 19% higher recall.

Broad first, careful second

The design is the lesson. GitHub did not ask one model to do everything:

  1. Rules cast a wide net. Simple patterns flag anything that looks like a secret. Fast, cheap, and tuned to over-flag rather than miss.
  2. A small model judges. It reads each candidate with its file path and nearby code, and filters out the lookalikes.

Each part does what it is best at. The rules give speed and coverage. The model gives judgement.

Three tests for a small model

GitHub concludes that a smaller fine-tuned model can be the right answer when the task is narrow, the decision is measurable, and efficiency is critical. As a checklist for your tasks:

  1. Is the task narrow? Yes or no, this or that. Not "write me something".
  2. Can you score it? You have labelled examples and can measure right and wrong.
  3. Does it run at volume? Thousands of times a day, where cost and speed add up.

Three yeses, and the big general model may be the expensive way to do a small job.

Do this today

  • Find your highest-volume AI task.
  • Check it against the three tests.
  • If it passes, collect a few hundred real examples with the right answers, including the confusing ones.
  • Test a small model against your current one on those examples before you change anything.

Your next step

I write one practical note like this every day. Sign up just below to get them by email.

Drafted with AI help, edited and approved by me.

New articles by email

I will never sell your details, for any reason, and I never send spam.

How I use your details
  • I, Abhijit Chatterjee Chattopadhyay of The Logic Atelier, hold them, in my own newsletter system hosted on Fly.io. Emails go out through Resend.
  • Your name is for greeting you, your email is for sending new articles, and your phone number, if you give it, is so I can reach you directly about my work.
  • If you do not confirm your email within 7 days, I delete the sign-up. If you unsubscribe, I delete your details 30 days later.
  • If Google Analytics is running on your visit, it is told that a sign-up was made, and nothing you typed.
  • To see or delete what I hold about you at any time, email hello@thelogicatelier.com.
  • The full detail is on the privacy page.
NextApproving everything means catching nothing
← cd ../writing

The small model that beat the LLM

small-model-beat-llm.mdfront matter
title: The small model that beat the LLM
date: 2026-10-20
reading_time: 2 min
author: Abhijit Chatterjee Chattopadhyay
small_model_beat_llm_cover_2b10c956_5156cf1a8f.pngcover
The Logic Atelier logo above the words: The small model that beat the LLM.

GitHub took a large language model out of one of its security tools and put a much smaller one in. Developers accepted more of its alerts, and serving it costs over 98% less.

What GitHub changed

GitHub's secret scanning looks for passwords and keys left in code. The hard part is the lookalikes: a test value, a placeholder in a guide, a checksum. Flag those and developers stop trusting the alerts.

According to GitHub's write-up (opens in a new tab), it replaced its previous LLM-based pipeline with a small classifier fine-tuned from ModernBERT. In its customer test against the old pipeline:

  • 11.5% higher acceptance of its alerts.
  • 86% lower median scan time after code is pushed.
  • More than 98% lower serving cost.

Offline, on public traffic, it also reported 13% higher precision and 19% higher recall.

Broad first, careful second

The design is the lesson. GitHub did not ask one model to do everything:

  1. Rules cast a wide net. Simple patterns flag anything that looks like a secret. Fast, cheap, and tuned to over-flag rather than miss.
  2. A small model judges. It reads each candidate with its file path and nearby code, and filters out the lookalikes.

Each part does what it is best at. The rules give speed and coverage. The model gives judgement.

Three tests for a small model

GitHub concludes that a smaller fine-tuned model can be the right answer when the task is narrow, the decision is measurable, and efficiency is critical. As a checklist for your tasks:

  1. Is the task narrow? Yes or no, this or that. Not "write me something".
  2. Can you score it? You have labelled examples and can measure right and wrong.
  3. Does it run at volume? Thousands of times a day, where cost and speed add up.

Three yeses, and the big general model may be the expensive way to do a small job.

Do this today

  • Find your highest-volume AI task.
  • Check it against the three tests.
  • If it passes, collect a few hundred real examples with the right answers, including the confusing ones.
  • Test a small model against your current one on those examples before you change anything.

Your next step

I write one practical note like this every day. Sign up just below to get them by email.

Drafted with AI help, edited and approved by me.

subscribeemail

// I will never sell your details, for any reason, and I never send spam.

How I use your details
  • I, Abhijit Chatterjee Chattopadhyay of The Logic Atelier, hold them, in my own newsletter system hosted on Fly.io. Emails go out through Resend.
  • Your name is for greeting you, your email is for sending new articles, and your phone number, if you give it, is so I can reach you directly about my work.
  • If you do not confirm your email within 7 days, I delete the sign-up. If you unsubscribe, I delete your details 30 days later.
  • If Google Analytics is running on your visit, it is told that a sign-up was made, and nothing you typed.
  • To see or delete what I hold about you at any time, email hello@thelogicatelier.com.
  • The full detail is on the privacy page.

vi.Contact

06 · contact

Talk to me.

Talk to me.

Production AI and software architecture. Based in Dubai, working with teams in the Emirates and Ireland.

I work Dubai hours, Monday to Friday, the same week you do. Dubai is three hours ahead of Dublin in summer and four in winter, so my afternoon is your morning.

I will never sell your details, for any reason, and I never send spam.

How I use your enquiry
  • Your enquiry comes to me, Abhijit Chatterjee Chattopadhyay of The Logic Atelier. The form is handled by Formspree, which passes it to my inbox and keeps a copy in my Formspree account.
  • I use it to read your enquiry and reply. If no work follows, I delete it within 12 months, and sooner whenever you ask.
  • If Google Analytics is running on your visit, it is told that an enquiry was sent, and nothing you typed.
  • To see or delete what I hold about you, email hello@thelogicatelier.com.
  • The full detail is on the privacy page.