Information-Theoretic Generalization Bounds for Deep Neural Networks

arXiv:2404.03176v1 Announce Type: new
Abstract: Deep neural networks (DNNs) exhibit an exceptional capacity for generalization in practical applications. This work aims to capture the effect and benefits of depth for supervised learning via information-theoretic generalization bounds. We first derive two hierarchical bounds on the generalization error in terms of the Kullback-Leibler (KL) divergence or the 1-Wasserstein distance between the train and test distributions of the network internal representations. The KL divergence bound shrinks as the layer index increases, while the …

abstract applications arxiv benefits capacity cs.it cs.lg divergence error hierarchical information math.it networks neural networks practical supervised learning terms type via work