Geometry-induced Implicit Regularization in Deep ReLU Neural Networks

It is well known that neural networks with many more parameters than training examples do not overfit. Implicit regularization phenomena, which are still not well understood, occur during optimization and 'good' networks are favored. Thus the number of parameters is not an adequate measure of complexity if we do not consider all possible networks but only the 'good' ones. To better understand which networks are favored during optimization, we study the geometry of the output set as parameters vary. When …

complexity cs.ai cs.lg cs.ne examples geometry good math.oc math.st networks neural networks optimization parameters regularization relu stat.th training