Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

arXiv:2402.13064v1 Announce Type: new
Abstract: We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human …

abstract arxiv construct cs.cl data datasets examples general generalized human language language models large language large language models llms prior scalable scratch seed synthetic synthetic data taxonomy type work