AWS has published a dual-monitoring design for production AI agents that separates answer quality from infrastructure health. AgentCore Evaluations samples live interactions and scores qualities such as helpfulness, correctness and goal completion, while AWS DevOps Agent investigates logs, traces and permissions when the underlying system fails.
The distinction matters because healthy service metrics do not prove that an agent completed a task. A supervisor can quietly route requests to the wrong specialist, or a booking agent can return nothing after a permission failure several calls deep. AWS demonstrates the setup with a four-agent airline reservation system whose specialists search flights, retrieve customer benefits and create bookings.
In one test, the DevOps agent traced a blank response to a missing permission to invoke the Bedrock model, following the path from the user request through AgentCore to the denied API call. AgentCore Evaluations can asynchronously sample from 0.01 to 100 percent of traces without adding user-facing latency. That sampling also defines a limit: a bad response may reach someone before it is scored. AWS recommends inline Bedrock Guardrails for immediate controls and says teams should treat model-judged evaluation scores as signals that require calibration, not objective ground truth.