Large language models can follow many individual instructions, but their performance degrades sharply when too many constraints must be satisfied at once, according to a new arXiv paper. The authors introduce Constraint Saturation Evaluation, a procedurally generated benchmark.
The benchmark varies the number of simultaneous constraints from one to twelve and scores outputs with deterministic rule-based checkers instead of model judges. Across 15 models, 36 constraint types and 369,753 checks, the paper reports three phase-transition-like regimes as constraints accumulate.
That matters for real products because agent prompts, safety rules and output schemas often stack many requirements into one interaction. A model that appears reliable on each rule separately may still fail when the rules are combined.
The study is a benchmark paper, so its practical impact depends on whether the generated tasks match production workloads. Its core warning is still useful: instruction following should be tested compositionally, not only one rule at a time.