Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimization

arXiv:2306.09222v3 Announce Type: replace
Abstract: We present Re-weighted Gradient Descent (RGD), a novel optimization technique that improves the performance of deep neural networks through dynamic sample importance weighting. Our method is grounded in the principles of distributionally robust optimization (DRO) with Kullback-Leibler divergence. RGD is simple to implement, computationally efficient, and compatible with widely used optimizers such as SGD and Adam. We demonstrate the broad applicability and impact of RGD by achieving state-of-the-art results on diverse benchmarks, including improvements of …

abstract arxiv cs.ai cs.lg divergence dynamic gradient importance networks neural networks novel optimization performance robust sample simple stochastic through type via