MAmmoTH2: Scaling Instructions from the Web

arXiv:2405.03548v1 Announce Type: new
Abstract: Instruction tuning improves the reasoning abilities of large language models (LLMs), with data quality and scalability being the crucial factors. Most instruction tuning data come from human crowd-sourcing or GPT-4 distillation. We propose a paradigm to efficiently harvest 10 million naturally existing instruction data from the pre-training web corpus to enhance LLM reasoning. Our approach involves (1) recalling relevant documents, (2) extracting instruction-response pairs, and (3) refining the extracted pairs using open-source LLMs. Fine-tuning base …

abstract arxiv crowd-sourcing cs.cl data data quality distillation gpt gpt-4 human instruction tuning language language models large language large language models llms paradigm pre-training quality reasoning scalability scaling training type web