AWS has published a nine-stage evaluation framework for AI agents that follows them from an initial code change through live production. The approach is designed to catch “soft” failures: answers that arrive on time and sound confident but contain wrong numbers, unsupported calculations or misleading labels.
The framework grew from an internal sales system built with Amazon Bedrock AgentCore. Several specialized agents answer questions about revenue, customers and underlying datasets while respecting each employee’s permissions. Conventional monitoring showed the service was responding normally, yet evaluation found serious content errors. One answer turned $410.08 into $410.08 million, another mislabeled a revenue figure, and another supplied an annual recurring revenue estimate that was absent from the source data.
AWS’s proposed quality loop tests individual components, complete workflows and production interactions instead of relying on a final pre-release check. The goal is to preserve test cases from real failures and rerun them automatically as prompts, tools, models and data change.
This is an implementation pattern rather than proof that AgentCore eliminates hallucinations. Its practical lesson is that latency and availability cannot measure whether an agent’s answer is correct. Teams still need domain-specific expected results, permissions tests and human review for consequential decisions.