LangSmith Engine v2 can now search for agent failures that have not yet appeared in production. Its Red Teaming feature studies an agent’s traces and code repository, generates relevant attack cases and verifies which weaknesses are real before presenting them to developers.
The system also broadens routine monitoring beyond explicit errors and unanswered requests. It looks for changes in latency, cost and error rates, as well as inefficient trajectories such as repeated tool calls or needlessly long reasoning paths. LangChain says the earlier Engine release analyzed more than 70 million traces and diagnosed tens of thousands of issues.
For agents hosted on LangSmith Deployment, Engine can reproduce a failing input and test candidate prompt or code changes against a wider evaluation set. A fix reaches the developer only after those automated checks, and a person still decides whether to open and merge the resulting pull request.
Red Teaming is currently a private beta for existing LangSmith Deployment customers. Generated tests remain dependent on the evidence and assumptions available to Engine, so teams still need their own security review and evaluation sets for high-impact behavior.