Compelling ReLU Network Initialization and Training to Leverage Exponential Scaling with Depth

A neural network with ReLU activations may be viewed as a composition of piecewise linear functions. For such networks, the number of distinct linear regions expressed over the input domain has the potential to scale exponentially with depth, but it is not expected to do so when the initial parameters are chosen randomly. This poor scaling can necessitate the use of overly large models to approximate even simple functions. To address this issue, we introduce a novel training strategy: we …

cs.ai cs.lg domain functions linear network networks neural network relu scale scaling training