Understanding Self-Distillation and Partial Label Learning in Multi-Class Classification with Label Noise

arXiv:2402.10482v1 Announce Type: new
Abstract: Self-distillation (SD) is the process of training a student model using the outputs of a teacher model, with both models sharing the same architecture. Our study theoretically examines SD in multi-class classification with cross-entropy loss, exploring both multi-round SD and SD with refined teacher outputs, inspired by partial label learning (PLL). By deriving a closed-form solution for the student model's outputs, we discover that SD essentially functions as label averaging among instances with high feature …

abstract architecture arxiv class classification cross-entropy cs.lg distillation entropy loss noise process stat.ml study training type understanding