A support agent can sound helpful and still make the same mistake tomorrow. What if its previous attempts became a better set of instructions? On October 20, IBM researchers described ALTK-Evolve doing that with GPT-4.1: extracting guidelines from training runs and adding them to the agent’s prompt, without changing the model’s weights.
In IBM’s reported test on 40 held-out retail tasks, the share passed on all four attempts rose from 57.5% to 67.5%. The evaluation used simulated customers and checked the resulting database state. Passing once and succeeding consistently are different questions; this measure asks the harder one.
The gains came with a tradeoff. IBM reports higher input-token costs, and a larger playbook was not always the most reliable choice across domains. We have not reproduced the results. They are research findings, rather than evidence that a particular support desk could safely automate more work.
The linked retail leaderboard submission adds another qualification: its full evaluation includes training tasks, and it expects an Unverified label because the prompt was modified. That submission is separate from the held-out test above. Read a leaderboard position alongside the test conditions, rather than treating it as a purchasing verdict.
Agentic AI
