A new paper proposes a data-efficient method for converting autoregressive language-model behavior into diffusion-style language models through on-policy distillation.
Diffusion language models are attracting interest because they may offer different tradeoffs in generation, refinement, and parallelism than standard next-token models.
The contribution is a training bridge: using an existing autoregressive policy to make diffusion language modeling more practical without starting from scratch.