Frequency Attention for Knowledge Distillation

arXiv:2403.05894v1 Announce Type: new
Abstract: Knowledge distillation is an attractive approach for learning compact deep neural networks, which learns a lightweight student model by distilling knowledge from a complex teacher model. Attention-based knowledge distillation is a specific form of intermediate feature-based knowledge distillation that uses attention mechanisms to encourage the student to better mimic the teacher. However, most of the previous attention-based distillation approaches perform attention in the spatial domain, which primarily affects local regions in the input image. This …

abstract arxiv attention attention mechanisms cs.cv distillation feature form intermediate knowledge networks neural networks type