How to escape sharp minima with random perturbations

Modern machine learning applications have witnessed the remarkable success of optimization algorithms that are designed to find flat minima. Motivated by this design choice, we undertake a formal study that (i) formulates the notion of flat minima, and (ii) studies the complexity of finding them. Specifically, we adopt the trace of the Hessian of the cost function as a measure of flatness, and use it to formally define the notion of approximate flat minima. Under this notion, we then analyze …

algorithms applications complexity cs.ai cs.lg design machine machine learning machine learning applications math.oc modern notion optimization random studies study success them