Amazon researchers have released HOPE, a pruning method that reduces the memory used by mixture-of-experts language models while accounting for experts that work together. These models activate only a small subset of their specialist feed-forward networks for each token, but all experts normally remain in memory during inference.
Common pruning methods score each expert independently and remove the lowest-ranked ones. HOPE instead measures pairwise co-activation and output strength on a calibration dataset, then solves an optimization problem intended to preserve useful groups. Calibration still requires one forward pass, while the added optimization reportedly takes one or two seconds per layer.
The team tested HOPE on Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air with two calibration sets and pruning rates from 10% to 50%. Across the reported conditions, HOPE had the best average rank among five methods. At 50% pruning its average rank was 1.58, compared with 2.42 for the next method. On Qwen3.5-35B, a half-pruned model scored 42.3 on SWE-Bench Pro versus 41.8 unpruned and 36.2 with REAP.
Those results come from the authors’ benchmarks and do not guarantee gains on unrelated workloads. HOPE’s advantage shrank at low pruning rates, it treats layers independently and captures pairs rather than larger expert groups. The code supports several Qwen and GLM families and outputs an ordinary smaller Hugging Face checkpoint.