Improve Generalization Ability of Deep Wide Residual Network with A Suitable Scaling Factor

arXiv:2403.04545v1 Announce Type: new
Abstract: Deep Residual Neural Networks (ResNets) have demonstrated remarkable success across a wide range of real-world applications. In this paper, we identify a suitable scaling factor (denoted by $\alpha$) on the residual branch of deep wide ResNets to achieve good generalization ability. We show that if $\alpha$ is a constant, the class of functions induced by Residual Neural Tangent Kernel (RNTK) is asymptotically not learnable, as the depth goes to infinity. We also highlight a surprising …

abstract alpha applications arxiv cs.lg good identify math.st network networks neural networks paper residual scaling stat.th success type world