Snorkel is running millions of agent simulations each month on Modal, using both CPU and GPU sandboxes to test coding, infrastructure and machine-learning agents in realistic environments. The setup lets evaluators distinguish a model failure from a missing tool or an underpowered test environment.
Demand arrives in sharp bursts. Snorkel can create tens of thousands of sandboxes in an hour when a new or tuned model needs fast evaluation. Some tests require a GPU, while infrastructure tasks may need a full Linux kernel or several connected services. Modal also hosts open-weight and customer-tuned models for the inference side of those tests.
The company has started using elastic “bursting,” which lets a sandbox temporarily consume more CPU or memory than its initial allocation. About 10% to 15% of its sandboxes use the feature so far. Snorkel says fewer runs fail from resource limits, while lower starting allocations have reduced costs by roughly 20%.
The case study shows why agent benchmarks increasingly resemble production systems: credible results depend not only on the model, but also on reproducible tools, compute and operating environments.