Weight-Inherited Distillation for Task-Agnostic BERT Compression

arXiv:2305.09098v2 Announce Type: replace-cross
Abstract: Knowledge Distillation (KD) is a predominant approach for BERT compression. Previous KD-based methods focus on designing extra alignment losses for the student model to mimic the behavior of the teacher model. These methods transfer the knowledge in an indirect way. In this paper, we propose a novel Weight-Inherited Distillation (WID), which directly transfers knowledge from the teacher. WID does not require any additional alignment loss and trains a compact student by inheriting the weights, showing …

arxiv bert compression cs.cl cs.lg distillation type