Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation

arXiv:2311.08640v3 Announce Type: replace-cross
Abstract: We study semi-supervised sequence generation tasks, where the few labeled examples are too scarce to finetune a model, and meanwhile, few-shot prompted large language models (LLMs) exhibit room for improvement. In this paper, we present the discovery that a student model distilled from a few-shot prompted LLM can commonly generalize better than its teacher to unseen examples on such tasks. We find that the student is able to learn a general pattern from the high-quality …

abstract arxiv collaborative cs.cl cs.lg discovery distillation examples few-shot improvement knowledge language language model language models large language large language model large language models llms paper room semi-supervised study tasks type