A simple connection from loss flatness to compressed representations in neural networks

arXiv:2310.01770v3 Announce Type: replace
Abstract: The generalization capacity of deep neural networks has been studied in a variety of ways, including at least two distinct categories of approaches: one based on the shape of the loss landscape in parameter space, and the other based on the structure of the representation manifold in feature space (that is, in the space of unit activities). Although these two approaches are related, they are rarely studied together explicitly. Here, we present an analysis that …

abstract arxiv capacity cs.ai cs.lg landscape least loss networks neural networks replace shape simple space type