Text Quality-Based Pruning for Efficient Training of Language Models

arXiv:2405.01582v1 Announce Type: cross
Abstract: In recent times training Language Models (LMs) have relied on computationally heavy training over massive datasets which makes this training process extremely laborious. In this paper we propose a novel method for numerically evaluating text quality in large unlabelled NLP datasets in a model agnostic manner to assign the text instances a "quality score".
By proposing the text quality metric, the paper establishes a framework to identify and eliminate low-quality text instances, leading to improved …

abstract arxiv cs.ai cs.cl cs.lg datasets language language models lms massive nlp novel paper process pruning quality text training type