Apple researchers have trained a three-billion-parameter masked diffusion model from scratch across text, image-text and audio-text data. Unlike an autoregressive model that produces tokens strictly from left to right, masked diffusion repeatedly fills and refines missing pieces, allowing a single design to support several kinds of generation.
The team calls this the first tri-modal masked diffusion model pretrained from the beginning on all three data mixtures. Its study examines how performance changes with model scale, the balance among modalities, noise schedules and batch size. The preliminary model was trained on 6.4 trillion tokens and produced results in text generation, text-to-image and text-to-speech tasks.
One contribution is a reparameterization based on a stochastic differential equation. It separates the physical batch size chosen for GPU utilization and training speed from the logical batch size that controls gradient variation. The researchers say this removes the need to search repeatedly for an optimal hardware batch size reported in earlier work.
The publication presents a systematic research study rather than a consumer model or product release. “Strong results” is the authors’ characterization, and the short public abstract does not provide enough benchmark detail to compare the model broadly with leading autoregressive or multimodal systems. Its immediate value is a set of scaling observations and inference defaults for researchers exploring discrete diffusion across multiple media.