Generalized Knowledge Distillation via Relationship Matching. (arXiv:2205.01915v1 [cs.CV])

Web: http://arxiv.org/abs/2205.01915

The knowledge of a well-trained deep neural network (a.k.a. the "teacher") is
valuable for learning similar tasks. Knowledge distillation extracts knowledge
from the teacher and integrates it with the target model (a.k.a. the
"student"), which expands the student's knowledge and improves its learning
efficacy. Instead of enforcing the teacher to work on the same task as the
student, we borrow the knowledge from a teacher trained from a general label
space -- in this "Generalized Knowledge Distillation (GKD)", the classes …

arxiv cv distillation knowledge relationship