Accurate Knowledge Distillation with n-best Reranking

arXiv:2305.12057v3 Announce Type: replace
Abstract: We propose utilizing n-best reranking to enhance Sequence-Level Knowledge Distillation (Kim and Rush, 2016) where we extract pseudo-labels for student model's training data from top n-best hypotheses and leverage a diverse set of models with different inductive biases, objective functions or architectures, including some publicly-available large language models, to pick the highest-quality hypotheses as labels. The effectiveness of our proposal is validated through experiments on the WMT'21 German-English and Chinese-English translation tasks. Our results demonstrate …

abstract architectures arxiv biases cs.cl data distillation diverse extract functions inductive knowledge labels language language models large language large language models set training training data type