GOVERN: Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillation

arXiv:2405.03764v1 Announce Type: new
Abstract: Pre-trained language models have become an integral component of question-answering systems, achieving remarkable performance. For practical deployment, it is critical to carry out knowledge distillation to preserve high performance under computational constraints. In this paper, we address a key question: given the importance of unsupervised distillation for student performance, how does one effectively ensemble knowledge from multiple teachers at this stage without the guidance of ground-truth labels? We propose a novel algorithm, GOVERN, to tackle …

abstract arxiv become computational constraints cs.cl cs.ir deployment distillation ensemble gradient importance integral key knowledge language language models paper performance practical question systems type unsupervised vote