GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation

arXiv:2403.19754v1 Announce Type: new
Abstract: Knowledge distillation from LLMs is essential for the efficient deployment of language models. Prior works have proposed data generation using LLMs for preparing distilled models. We argue that generating data with LLMs is prone to sampling mainly from the center of original content distribution. This limitation hinders the distilled model from learning the true underlying data distribution and to forget the tails of the distributions (samples with lower probability). To this end, we propose GOLD, …

abstract arxiv center cs.cl data deployment distillation distribution generalized knowledge language language data language models llms prior sampling type via