Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information

arXiv:2401.08732v2 Announce Type: replace
Abstract: It is believed that in knowledge distillation (KD), the role of the teacher is to provide an estimate for the unknown Bayes conditional probability distribution (BCPD) to be used in the student training process. Conventionally, this estimate is obtained by training the teacher using maximum log-likelihood (MLL) method. To improve this estimate for KD, in this paper we introduce the concept of conditional mutual information (CMI) into the estimation of BCPD and propose a novel …

arxiv bayes cs.cv cs.it cs.lg distillation distribution information knowledge math.it type