Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

arXiv:2403.16952v1 Announce Type: cross
Abstract: Pretraining data of large language models composes multiple domains (e.g., web texts, academic papers, codes), whose mixture proportions crucially impact the competence of outcome models. While existing endeavors rely on heuristics or qualitative strategies to tune the proportions, we discover the quantitative predictability of model performance regarding the mixture proportions in function forms, which we refer to as the data mixing laws. Fitting such functions on sample mixtures unveils model performance on unseen mixtures before …

abstract academic arxiv cs.ai cs.cl cs.lg data domains heuristics impact language language models large language large language models laws modeling multiple papers performance pretraining quantitative strategies type web