The loss landscape of deep linear neural networks: a second-order analysis. (arXiv:2107.13289v2 [math.ST] CROSS LISTED)

We study the optimization landscape of deep linear neural networks with the
square loss. It is known that, under weak assumptions, there are no spurious
local minima and no local maxima. However, the existence and diversity of
non-strict saddle points, which can play a role in first-order algorithms'
dynamics, have only been lightly studied. We go a step further with a full
analysis of the optimization landscape at order 2. We characterize, among all
critical points, which are global minimizers, …

analysis arxiv landscape loss math networks neural networks