AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation

arXiv:2211.10938v2 Announce Type: replace-cross
Abstract: We present a novel adversarial penalized self-knowledge distillation method, named adversarial learning and implicit regularization for self-knowledge distillation (AI-KD), which regularizes the training procedure by adversarial learning and implicit distillations. Our model not only distills the deterministic and progressive knowledge which are from the pre-trained and previous epoch predictive probabilities but also transfers the knowledge of the deterministic predictive distributions using adversarial learning. The motivation is that the self-knowledge distillation methods regularize the predictive probabilities …

abstract adversarial adversarial learning arxiv cs.cv cs.lg distillation knowledge novel regularization training type