Researchers studied over 2,000 language-model training runs to understand the trade-off between mixing scarce target data (like low-resource languages or specialized domains) with abundant generic data during pretraining. The findings show repetition is the central driver of target-domain performance, and mixture training tolerates far higher repetition than training on a single source alone—scarce corpora can be reused 15-20 times depending on target data size, compute budget, and model scale. A new repetition-aware mixture scaling law is introduced that accounts for diminishing returns from repeated tokens and the regularizing effect of generic data, offering practical guidance for computing optimal mixture configurations under data constraints.
Questions this post answers
How many times can I repeat a small target dataset when mixing it with generic data for LLM pretraining?
Scarce target corpora can be reused 15 to 20 times when mixed with generic data during pretraining, tolerating far more repetition than single-source training would allow. The optimal repeat count depends on the target data size, compute budget, and model scale, based on analysis of over 2,000 language-model training runs across multilingual, domain-specific, and quality-filtered mixtures. Teams tuning pretraining mixtures for low-resource domains can track this kind of scaling research on daily.dev.