Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory

In this paper, we explain the universal approximation capabilities of deep residual neural networks through geometric nonlinear control. Inspired by recent work establishing links between residual networks and control systems, we provide a general sufficient condition for a residual network to have the power of universal approximation by asking the activation function, or one of its derivatives, to satisfy a quadratic differential equation. Many activation functions used in practice satisfy this assumption, exactly or approximately, and we show this property …

approximation capabilities control control systems cs.lg cs.sy eess.sy general math.oc network networks neural networks paper power residual stat.ml systems theory through via work