Sharp asymptotics on the compression of two-layer neural networks. (arXiv:2205.08199v4 [cs.IT] UPDATED)

In this paper, we study the compression of a target two-layer neural network
with N nodes into a compressed network with M<N nodes. More precisely, we
consider the setting in which the weights of the target network are i.i.d.
sub-Gaussian, and we minimize the population L_2 loss between the outputs of
the target and of the compressed network, under the assumption of Gaussian
inputs. By using tools from high-dimensional probability, we show that this
non-convex problem can be simplified when …

arxiv compression networks neural networks