Apple researchers have demonstrated a way to shrink the streaming neural audio encoder used before speech reaches a foundation model. The work targets on-device dictation, where an always-active speech tokenizer competes with the language model for memory and directly affects power use and response time.
The method trains a smaller student encoder to reproduce the teacher model’s internal representation immediately before quantization. Rather than copying discrete speech tokens or the model’s final output distribution, the student learns the latent representation shared by two token interfaces. A single affine layer handles the difference in width between teacher and student.
At 2.8-times compression, the distilled encoder remained within 1.9% relative word-error rate of its teacher in five of six teacher-student pairings, without fine-tuning. It also improved relative word-error rate by 3.9% compared with a separately trained tokenizer of the same capacity.
The result is research rather than a newly announced consumer feature. Its practical value is a smaller always-on component that leaves more device memory and power budget for the main model while preserving most transcription accuracy. The paper says the same recipe works with tokenizers trained independently or jointly with a language model.