On Measuring Excess Capacity in Neural Networks. (arXiv:2202.08070v2 [cs.LG] UPDATED)

We study the excess capacity of deep networks in the context of supervised
classification. That is, given a capacity measure of the underlying hypothesis
class -- in our case, empirical Rademacher complexity -- by how much can we (a
priori) constrain this class while retaining an empirical error on a par with
the unconstrained regime? To assess excess capacity in modern architectures
(such as residual networks), we extend and unify prior Rademacher complexity
bounds to accommodate function composition and addition, …

arxiv capacity lg networks neural networks