The small model that beat the LLM
GitHub replaced a large language model with a small one. It raised alert acceptance 11.5% and cut serving cost over 98%. When to do the same.
Abhijit Chatterjee Chattopadhyay

GitHub took a large language model out of one of its security tools and put a much smaller one in. Developers accepted more of its alerts, and serving it costs over 98% less.
What GitHub changed
GitHub's secret scanning looks for passwords and keys left in code. The hard part is the lookalikes: a test value, a placeholder in a guide, a checksum. Flag those and developers stop trusting the alerts.
According to GitHub's write-up (opens in a new tab), it replaced its previous LLM-based pipeline with a small classifier fine-tuned from ModernBERT. In its customer test against the old pipeline:
- 11.5% higher acceptance of its alerts.
- 86% lower median scan time after code is pushed.
- More than 98% lower serving cost.
Offline, on public traffic, it also reported 13% higher precision and 19% higher recall.
Broad first, careful second
The design is the lesson. GitHub did not ask one model to do everything:
- Rules cast a wide net. Simple patterns flag anything that looks like a secret. Fast, cheap, and tuned to over-flag rather than miss.
- A small model judges. It reads each candidate with its file path and nearby code, and filters out the lookalikes.
Each part does what it is best at. The rules give speed and coverage. The model gives judgement.
Three tests for a small model
GitHub concludes that a smaller fine-tuned model can be the right answer when the task is narrow, the decision is measurable, and efficiency is critical. As a checklist for your tasks:
- Is the task narrow? Yes or no, this or that. Not "write me something".
- Can you score it? You have labelled examples and can measure right and wrong.
- Does it run at volume? Thousands of times a day, where cost and speed add up.
Three yeses, and the big general model may be the expensive way to do a small job.
Do this today
- Find your highest-volume AI task.
- Check it against the three tests.
- If it passes, collect a few hundred real examples with the right answers, including the confusing ones.
- Test a small model against your current one on those examples before you change anything.
Your next step
I write one practical note like this every day. Sign up just below to get them by email.
Drafted with AI help, edited and approved by me.