It's All in the Head: Representation Knowledge Distillation through Classifier Sharing. (arXiv:2201.06945v2 [cs.CV] UPDATED)

Representation knowledge distillation aims at transferring rich information
from one model to another. Common approaches for representation distillation
mainly focus on the direct minimization of distance metrics between the models'
embedding vectors. Such direct methods may be limited in transferring
high-order dependencies embedded in the representation vectors, or in handling
the capacity gap between the teacher and student models. Moreover, in standard
knowledge distillation, the teacher is trained without awareness of the
student's characteristics and capacity. In this paper, we …

arxiv cv distillation head knowledge representation