Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation

arXiv:2403.01479v1 Announce Type: new
Abstract: The advent of scalable deep models and large datasets has improved the performance of Neural Machine Translation. Knowledge Distillation (KD) enhances efficiency by transferring knowledge from a teacher model to a more compact student model. However, KD approaches to Transformer architecture often rely on heuristics, particularly when deciding which teacher layers to distill from. In this paper, we introduce the 'Align-to-Distill' (A2D) strategy, designed to address the feature mapping problem by adaptively aligning student attention …

abstract alignment architecture arxiv attention cs.ai cs.cl datasets distillation efficiency knowledge large datasets machine machine translation neural machine translation performance scalable transformer transformer architecture translation type