EdgeXpert targets a hardware bottleneck in running large language models on edge devices: repeated external memory access in feed-forward layers. The paper combines mixture-of-experts, which activates only part of a model, with speculative decoding, which drafts multiple tokens to reduce decoding stages.
The authors argue those two techniques are not naturally compatible and propose a software-hardware co-design to make them work together. In the prefill stage, routing is reformulated around prompt-level expert reuse rather than independent token-level choices.
This is a research accelerator design, not a device consumers can buy today. Its relevance is clear, however: local AI depends on cutting memory traffic as much as raw compute, especially when models must run privately or with low latency.