Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models

arXiv:2402.10430v1 Announce Type: new
Abstract: Instruction-tuning language models has become a crucial step in aligning them for general use. Typically, this process involves extensive training on large datasets, incurring high training costs. In this paper, we introduce a novel training data selection based on the learning percentage of the samples. We assert that current language models possess the capability to autonomously select high-quality training data, leading to comparable or improved performance compared to training on the entire dataset. Our experiments …

abstract arxiv become costs cs.cl data datasets general language language models large datasets novel paper process them training training costs training data type