Researchers have built a benchmark for local language-model agents that operate hardware-design software through Model Context Protocol tools. The work targets companies that cannot send confidential component specifications or naming conventions to hosted AI services.
The team created an MCP server reproducing the state and dependency rules of a proprietary embedded-system design tool. Tasks include creating components, adding ports, wiring connections in the correct order, handling invalid requests and misspellings, and choosing tools when multiple servers are available.
Seven open-source models were tested with different system prompts, tool descriptions, context sizes, and single- or multi-agent setups. Strong models achieved near-complete coverage of expected calls on the benchmark, but results changed substantially with configuration. Detailed tool descriptions generally reduced failures, while few-shot examples caused severe inaction for some models.
Long cumulative context hurt constrained models. Splitting work among agents helped weaker models or longer sessions, but required additional calls. The benchmark emulates one proprietary workflow rather than proving readiness for chip design broadly. It provides practical evidence that local deployment is possible while showing that prompt design, context management, and dependency validation remain part of the engineering work.