High dimensional analysis reveals conservative sharpening and a stochastic edge of stability

arXiv:2404.19261v1 Announce Type: new
Abstract: Recent empirical and theoretical work has shown that the dynamics of the large eigenvalues of the training loss Hessian have some remarkably robust features across models and datasets in the full batch regime. There is often an early period of progressive sharpening where the large eigenvalues increase, followed by stabilization at a predictable value known as the edge of stability. Previous work showed that in the stochastic setting, the eigenvalues increase more slowly - a …

abstract analysis arxiv cs.lg datasets dynamics edge features loss math.oc math.st physics.data-an robust stability stat.th stochastic training training loss type work