Apple researchers have developed Agent Seer, a pipeline that creates evaluation scenarios for tool-using AI agents from a single Model Context Protocol specification. It needs no examples, live access to the tool, or domain-specific tuning.
The system reads function names, descriptions, and typed parameter schemas, then enriches those definitions and generates scenarios with synthetic tool outputs. It expands them into mock-data-grounded, multi-turn conversations designed to test whether an agent chooses the right tools and supplies appropriate arguments.
The researchers evaluated Agent Seer on seven MCP specifications spanning different domains and tool-suite sizes. They report strong tool-calling correctness and conversational coherence, with complete tool coverage for small and medium specifications. Complex parameter schemas were more closely associated with quality variation than the number of available tools.
Incorrect argument values were the largest failure mode among imperfect scenarios. That limitation matters because a benchmark can look successful when it checks only whether an agent selected the correct function. Agent Seer is a research method rather than proof that synthetic tests can replace human review, but it could help test suites keep pace with frequently changing APIs.