XGrad: Boosting Gradient-Based Optimizers With Weight Prediction

arXiv:2305.18240v2 Announce Type: replace
Abstract: In this paper, we propose a general deep learning training framework XGrad which introduces weight prediction into the popular gradient-based optimizers to boost their convergence and generalization when training the deep neural network (DNN) models. In particular, ahead of each mini-batch training, the future weights are predicted according to the update rule of the used optimizer and are then applied to both the forward pass and backward propagation. In this way, during the whole training …

arxiv boosting cs.ai cs.lg gradient prediction type