all AI news
Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information
March 11, 2024, 4:42 a.m. | Linfeng Ye, Shayan Mohajer Hamidi, Renhao Tan, En-Hui Yang
cs.LG updates on arXiv.org arxiv.org
Abstract: It is believed that in knowledge distillation (KD), the role of the teacher is to provide an estimate for the unknown Bayes conditional probability distribution (BCPD) to be used in the student training process. Conventionally, this estimate is obtained by training the teacher using maximum log-likelihood (MLL) method. To improve this estimate for KD, in this paper we introduce the concept of conditional mutual information (CMI) into the estimation of BCPD and propose a novel …
arxiv bayes cs.cv cs.it cs.lg distillation distribution information knowledge math.it type
More from arxiv.org / cs.LG updates on arXiv.org
Jobs in AI, ML, Big Data
Lead Developer (AI)
@ Cere Network | San Francisco, US
Research Engineer
@ Allora Labs | Remote
Ecosystem Manager
@ Allora Labs | Remote
Founding AI Engineer, Agents
@ Occam AI | New York
AI Engineer Intern, Agents
@ Occam AI | US
AI Research Scientist
@ Vara | Berlin, Germany and Remote