Sharp asymptotics on the compression of two-layer neural networks. (arXiv:2205.08199v2 [cs.IT] UPDATED)

In this paper, we study the compression of a target two-layer neural network
with N nodes into a compressed network with M < N nodes. More precisely, we
consider the setting in which the weights of the target network are i.i.d.
sub-Gaussian, and we minimize the population L2 loss between the outputs of the
target and of the compressed network, under the assumption of Gaussian inputs.
By using tools from high-dimensional probability, we show that this non-convex
problem can be …

arxiv compression networks neural networks