Sliding down the stairs: how correlated latent variables accelerate learning with neural networks

arXiv:2404.08602v1 Announce Type: cross
Abstract: Neural networks extract features from data using stochastic gradient descent (SGD). In particular, higher-order input cumulants (HOCs) are crucial for their performance. However, extracting information from the $p$th cumulant of $d$-dimensional inputs is computationally hard: the number of samples required to recover a single direction from an order-$p$ tensor (tensor PCA) using online SGD grows as $d^{p-1}$, which is prohibitive for high-dimensional inputs. This result raises the question of how neural networks extract relevant directions …

abstract arxiv cond-mat.stat-mech cs.lg data extract features gradient however information inputs math.pr math.st networks neural networks performance samples stat.ml stat.th stochastic type variables