AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting

arXiv:2405.08019v1 Announce Type: new
Abstract: Knowledge distillation, a widely used model compression technique, works on the basis of transferring knowledge from a cumbersome teacher model to a lightweight student model. The technique involves jointly optimizing the task specific and knowledge distillation losses with a weight assigned to them. Despite these weights playing a crucial role in the performance of the distillation process, current methods provide equal weight to both losses, leading to suboptimal performance. In this paper, we propose Adaptive …

abstract arxiv asr compression cs.ai cs.lg distillation dynamic knowledge loss losses them type