ReffAKD: Resource-efficient Autoencoder-based Knowledge Distillation

arXiv:2404.09886v1 Announce Type: new
Abstract: In this research, we propose an innovative method to boost Knowledge Distillation efficiency without the need for resource-heavy teacher models. Knowledge Distillation trains a smaller ``student'' model with guidance from a larger ``teacher'' model, which is computationally costly. However, the main benefit comes from the soft labels provided by the teacher, helping the student grasp nuanced class similarities. In our work, we propose an efficient method for generating these soft labels, thereby eliminating the need …

arxiv autoencoder cs.cv cs.lg distillation knowledge type