Researchers at Multiverse Computing have developed a way to remove whole transformer blocks from a language model while preserving more of its performance. Removing blocks makes a model shorter, delivering predictable inference speedups and memory savings, but choosing the wrong combination can severely damage its abilities.
The method treats each block as a binary choice — keep or remove — and models the interactions among choices as an Ising glass, a physics system of connected spins. A mathematical approximation of the model’s loss supplies an energy score for each candidate combination. Low-energy configurations tend to correspond to pruned models that perform better, allowing researchers to search many options without running a full benchmark on each one.
On Llama 3.3 70B Instruct with half its blocks removed, the team reports an MMLU score almost 23 percentage points above the best competing block-removal method it tested. MMLU is a broad academic knowledge benchmark, not a complete measure of usefulness. The researchers computed the required interaction matrix once from a small calibration dataset, then used brute force or a fast tabu-search solver to identify candidates.
The energy estimate is not perfect, so the lowest-energy configuration is not always the best actual model. The workflow therefore produces several promising candidates for real evaluation rather than claiming one mathematical answer guarantees quality.