AWS and Motorway have published a production blueprint for evaluating AI agents, based on work behind Motorway’s dealer stock search assistant. The UK car marketplace uses the agent to let dealers search vehicle listings with natural-language queries instead of manual filters.
The problem was reliability. A confident answer is not enough when dealers are making business decisions: tool-selection errors, misunderstood constraints, context drift, and nondeterministic outputs can all produce wrong search results. AWS says the evaluation pipeline reduced incorrect results from 1 in 8 queries to 1 in 50 and cut issue detection from hours to minutes.
The pipeline combines the Strands Agents SDK with Amazon Bedrock AgentCore, a managed service for deploying and operating agents. It tests realistic queries, tracks failures, and gives teams a repeatable way to measure agent behavior instead of relying on one-off demonstrations.
The broader lesson is that production agents need evaluation systems as much as they need better prompts. Without continuous measurement, teams may not notice when a useful assistant becomes unreliable in edge cases.