Technical Report: Competition Solution For BetterMixture

arXiv:2403.13233v1 Announce Type: new
Abstract: In the era of flourishing large-scale models, the challenge of selecting and optimizing datasets from the vast and complex sea of data, to enhance the performance of large language models within the constraints of limited computational resources, has become paramount. This paper details our solution for the BetterMixture challenge, which focuses on the fine-tuning data mixing for large language models. Our approach, which secured third place, incorporates data deduplication, low-level and high-level quality filtering, and …

abstract arxiv become challenge competition computational constraints cs.cl data datasets language language models large language large language models large-scale models paper performance report resources scale solution technical type vast