QuRating: Selecting High-Quality Data for Training Language Models

arXiv:2402.09739v1 Announce Type: cross
Abstract: Selecting high-quality pre-training data is important for creating capable language models, but existing methods rely on simple heuristics. We introduce QuRating, a method for selecting pre-training data that captures the abstract qualities of texts which humans intuitively perceive. In this paper, we investigate four qualities - writing style, required expertise, facts & trivia, and educational value. We find that LLMs are able to discern these qualities and observe that they are better at making pairwise …

abstract arxiv cs.cl cs.lg data heuristics humans language language models paper pre-training quality quality data simple training training data type writing