Knowledge Distillation Under Ideal Joint Classifier Assumption. (arXiv:2304.11004v1 [cs.LG])

Knowledge distillation is a powerful technique to compress large neural
networks into smaller, more efficient networks. Softmax regression
representation learning is a popular approach that uses a pre-trained teacher
network to guide the learning of a smaller student network. While several
studies explored the effectiveness of softmax regression representation
learning, the underlying mechanism that provides knowledge transfer is not well
understood. This paper presents Ideal Joint Classifier Knowledge Distillation
(IJCKD), a unified framework that provides a clear and comprehensive
understanding …

arxiv classifier distillation foundation framework guide knowledge network networks neural networks paper popular regression representation representation learning softmax studies transfer understanding