SADT: Combining Sharpness-Aware Minimization with Self-Distillation for Improved Model Generalization. (arXiv:2211.00310v1 [cs.LG])

Methods for improving deep neural network training times and model
generalizability consist of various data augmentation, regularization, and
optimization approaches, which tend to be sensitive to hyperparameter settings
and make reproducibility more challenging. This work jointly considers two
recent training strategies that address model generalizability: sharpness-aware
minimization, and self-distillation, and proposes the novel training strategy
of Sharpness-Aware Distilled Teachers (SADT). The experimental section of this
work shows that SADT consistently outperforms previously published training
strategies in model convergence time, test-time …

arxiv distillation model generalization