Global Convergence Rate of Deep Equilibrium Models with General Activations

arXiv:2302.05797v3 Announce Type: replace-cross
Abstract: In a recent paper, Ling et al. investigated the over-parametrized Deep Equilibrium Model (DEQ) with ReLU activation. They proved that the gradient descent converges to a globally optimal solution for the quadratic loss function at a linear convergence rate. This paper shows that this fact still holds for DEQs with any generally bounded activation with bounded first and second derivatives. Since the new activation function is generally non-homogeneous, bounding the least eigenvalue of the Gram …

abstract arxiv convergence cs.lg equilibrium function general global gradient linear loss paper rate relu shows solution stat.ml type