How to Train Data-Efficient LLMs

arXiv:2402.09668v1 Announce Type: new
Abstract: The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned …

abstract aim arxiv compute consumption cs.ai cs.cl cs.lg data language language models large language large language models llms paper pareto pre-training quality study train training training llms type