Contrastive Representation Distillation. (arXiv:1910.10699v3 [cs.LG] UPDATED)

Often we wish to transfer representational knowledge from one neural network
to another. Examples include distilling a large network into a smaller one,
transferring knowledge from one sensory modality to a second, or ensembling a
collection of models into a single estimator. Knowledge distillation, the
standard approach to these problems, minimizes the KL divergence between the
probabilistic outputs of a teacher and student network. We demonstrate that
this objective ignores important structural knowledge of the teacher network.
This motivates an …

arxiv distillation