Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks. (arXiv:2210.04834v3 [cs.CL] UPDATED)

Teacher-student knowledge distillation is a popular technique for compressing
today's prevailing large language models into manageable sizes that fit
low-latency downstream applications. Both the teacher and the choice of
transfer set used for distillation are crucial ingredients in creating a high
quality student. Yet, the generic corpora used to pretrain the teacher and the
corpora associated with the downstream target domain are often significantly
different, which raises a natural question: should the student be distilled
over the generic corpora, so …

arxiv distillation impact knowledge nlu transfer