IBM researchers report that a customer-service agent powered by GPT-4.1 became more reliable after receiving a learned playbook, without changing the model’s weights. The approach, called ALTK-Evolve, turns successes and failures on training tasks into short behavioral guidelines.
The rules cover actions such as authenticating customers, confirming transactions, and handling returns. They are appended to the system prompt of a standard ReAct agent, while the underlying tools, policies, and agent framework remain unchanged.
On held-out retail tasks from the τ²-bench benchmark, average success rose from 78.1% to 82.5%. The stricter Pass^4 score—the share of tasks completed correctly in all four independent attempts—increased from 57.5% to 67.5%. Airline-task average accuracy improved from 57.5% to 65%.
The benchmark uses simulated customers and checks whether the final database state matches the expected outcome. That is a demanding test of policy-following, but it is still not a live support operation. The result suggests that compact, experience-derived instructions can improve consistency when retraining or adopting a newer frontier model is impractical.