The Geometry of the Set of Equivalent Linear Neural Networks

arXiv:2404.14855v1 Announce Type: new
Abstract: We characterize the geometry and topology of the set of all weight vectors for which a linear neural network computes the same linear transformation $W$. This set of weight vectors is called the fiber of $W$ (under the matrix multiplication map), and it is embedded in the Euclidean weight space of all possible weight vectors. The fiber is an algebraic variety that is not necessarily a manifold. We describe a natural way to stratify the …

abstract arxiv cs.cg cs.lg embedded geometry linear map matrix matrix multiplication network networks neural network neural networks set the matrix topology transformation type vectors