Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth

Autoencoders are a prominent model in many empirical branches of machine learning and lossy data compression. However, basic theoretical questions remain unanswered even in a shallow two-layer setting. In particular, to what degree does a shallow autoencoder capture the structure of the underlying data distribution? For the prototypical case of the 1-bit compression of sparse Gaussian data, we prove that gradient descent converges to a solution that completely disregards the sparse structure of the input. Namely, the performance of the …

autoencoder autoencoders basic benefit compression cs.it cs.lg data data compression distribution layer machine machine learning math.it questions stat.ml structured data