Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation

arXiv:2404.14827v1 Announce Type: new
Abstract: Knowledge distillation, transferring knowledge from a teacher model to a student model, has emerged as a powerful technique in neural machine translation for compressing models or simplifying training targets. Knowledge distillation encompasses two primary methods: sentence-level distillation and token-level distillation. In sentence-level distillation, the student model is trained to align with the output of the teacher model, which can alleviate the training difficulty and give student model a comprehensive understanding of global structure. Differently, token-level …

abstract arxiv cs.cl distillation knowledge machine machine translation neural machine translation simplifying study targets token training translation type