Annealing Self-Distillation Rectification Improves Adversarial Training

arXiv:2305.12118v2 Announce Type: replace
Abstract: In standard adversarial training, models are optimized to fit one-hot labels within allowable adversarial perturbation budgets. However, the ignorance of underlying distribution shifts brought by perturbations causes the problem of robust overfitting. To address this issue and enhance adversarial robustness, we analyze the characteristics of robust models and identify that robust models tend to produce smoother and well-calibrated outputs. Based on the observation, we propose a simple yet effective method, Annealing Self-Distillation Rectification (ADR), which …

abstract adversarial adversarial training analyze arxiv budgets cs.ai cs.lg distillation distribution hot however identify issue labels overfitting robust robust models robustness standard training type