DoGE: Domain Reweighting with Generalization Estimation

The coverage and composition of the pretraining data significantly impacts the generalization ability of Large Language Models (LLMs). Despite its importance, recent LLMs still rely on heuristics and trial and error to increase or reduce the influence of data-domains. We propose DOmain reweighting with Generalization Estimation (DoGE), which optimizes the probability of sampling from each domain (domain weights) in a principled way. Our approach is a two-stage process consisting of (i) training a proxy model to obtain domain weights using …

coverage cs.ai cs.cl cs.lg data domain domains error heuristics impacts importance influence language language models large language large language models llms pretraining probability reduce sampling