Researchers have proposed X-CoSD, a framework for faster language-model inference across an end device and an edge server. In collaborative speculative decoding, a small local model drafts several tokens and a larger server model verifies them, potentially producing the same output faster than the server model alone.
Existing systems generally assume both models use the same vocabulary. They can also transmit large probability distributions during residual resampling, the correction process used when the larger model rejects a draft. X-CoSD supports different vocabularies and divides that correction work between the device and server. It sends probability data only for vocabulary shared by both models.
An enhanced version, X-CoSD-E, reduces communication further. The server samples replacement candidates and sends only those candidates and their probabilities to the device for verification, rather than transmitting a full distribution. The authors describe both approaches as lossless, meaning they preserve the larger model’s output distribution rather than approximating it for speed.
The design could make hybrid on-device inference practical on constrained networks and allow independently trained small and large models to cooperate. The paper is a new preprint, so its reported efficiency gains still need replication and evaluation on production hardware and real network conditions.