Apple researchers report that adding data from a high-resource language can be more effective than extensive hyperparameter tuning when training a language model for a language with limited text. Their experiments used Arabic as the target language and English as the auxiliary source.
The team ran roughly 1,000 pretraining experiments across models ranging from 150 million to 1.43 billion parameters. Mixing languages produced larger gains than tuning settings such as weight decay on both validation loss and downstream task accuracy, and the advantage increased with model size.
According to the paper, the improvement from mixing was comparable to having two to three times as much unique Arabic data when measured by validation loss. On downstream tasks, it was equivalent to two to 13 times as much target-language data. The authors argue that ordinary target-language validation loss understates the benefit because auxiliary text supplies knowledge as well as regularization.
The result comes from one bilingual pairing and a controlled research setup, so it does not establish the best mixture for every language. The practical recommendation is to prioritize the language-mixing ratio, then transfer hyperparameters from a smaller proxy model rather than repeating an expensive search at full scale.