Right three times out of four is not reliable
Average accuracy hides the people it lets down. An IBM Research team raised every-time success with short learned rules instead of a bigger model.
Abhijit Chatterjee Chattopadhyay

An AI support agent that handles a refund correctly three times out of four sounds good. As an IBM Research team put it this week, that agent "still fails a quarter of real customers".
The line comes from their write-up on ALTK-Evolve (opens in a new tab), published on 20 October. It is the right way to think about agents.
Average is not the number
Most AI demos report accuracy: how often the agent gets a task right, on average. The IBM team also reports something stricter, called Pass^4. Every task runs four times, and it only counts if all four attempts succeed.
On their retail customer-service tests, a basic agent averaged 78.1% accuracy. By the stricter measure it scored 57.5%. Same agent, same tasks. The gap between the two numbers is the customers who get a different result depending on when they ask.
A playbook, not a bigger model
What closed part of that gap was not a new model. The team used GPT-4.1, which they call an older, non-reasoning model, and changed one thing. The model reviewed the agent's past attempts, successes and failures, and wrote short rules from them. Those rules were added to the agent's instructions.
The rules are plain. Two from their list:
- "Authenticate the user before taking any order-related action. If email is unavailable, verify identity via full name and zip code using the find_user_id_by_name_zip API."
- "Always confirm all intended changes (modify, cancel, return, exchange) with a clear, structured summary, and require explicit user approval before executing any database-updating action."
On tasks the agent had not seen before, the stricter score rose from 57.5% to 67.5% in retail, and from 35% to 45% in airline bookings. In retail, that put it above every entry in the benchmark's leaderboard as the team read it on 12 October. The rules are not only a patch for an older model either: when the team reran the airline tests with the newer GPT-5.5, its every-time score rose from 55% to 70%.
Two caveats. This is a benchmark with simulated customers and small test sets (40 retail and 20 airline tasks), not live traffic, and the results are the team's own. And it is not free: in retail, the longer instructions raised the cost per conversation by about 65%. The team notes that prompt caching can win much of that back.
More rules is not always better
One finding I would not skip. In retail, keeping every learned rule, all 292 of them, worked best. In airline, a short core of 14 rules did slightly better on the every-time measure than the full set of 101 (45% against 40%, better by one task out of twenty), though the full set scored higher on average.
How much guidance helps depends on the job. Test it, do not assume it.
Do this today
- For your most important agent task, run the same test cases several times, not once.
- Report two numbers: average success, and the share of cases that succeed every time.
- Collect the failures and write each lesson as one short rule in the agent's instructions.
- Re-run the same cases. Keep the rules that move the every-time number, and drop the ones that do not.
Your next step
I write one practical note like this every day. Sign up just below to get them by email.
Drafted with AI help, edited and approved by me.