A controlled study involving researchers from Princeton University, UC San Diego, and other institutions examined why stored “skills” help AI agents. Across 8,135 test runs, agents completed identical tasks with and without compact instruction sets covering steps, checks, tools, and common mistakes.
The researchers found that procedural grounding explained 65.7% of cases in which a skilled agent outperformed an unassisted one. Added knowledge accounted for only 4.5%. Skills were most useful as playbooks: they helped agents configure environments, choose tools in sequence, perform intermediate checks, and produce the required output format. Related skills could also provide useful direction even without an exact match.
The approach introduced its own failure modes. In 10% of cases, agents applied a useful playbook too mechanically or used it where it did not fit. Retrieval also deteriorated sharply as the library expanded. When the available collection grew from five skills to 100, precision in selecting the right one fell from 29.6% to 3.3%, with similarly named options making selection more difficult.
The study suggests that simply storing more past experience will not necessarily create better self-learning agents. More reliable systems will need to manage the full skill lifecycle: creating useful procedures, retrieving the relevant ones, and applying them selectively rather than treating every instruction set as universally appropriate.