Self-Knowledge Distillation via Dropout. (arXiv:2208.05642v1 [cs.CV])

To boost the performance, deep neural networks require deeper or wider
network structures that involve massive computational and memory costs. To
alleviate this issue, the self-knowledge distillation method regularizes the
model by distilling the internal knowledge of the model itself. Conventional
self-knowledge distillation methods require additional trainable parameters or
are dependent on the data. In this paper, we propose a simple and effective
self-knowledge distillation method using a dropout (SD-Dropout). SD-Dropout
distills the posterior distributions of multiple models through a …

arxiv cv distillation dropout knowledge