Polysemanticity and Capacity in Neural Networks. (arXiv:2210.01892v1 [cs.NE])

Individual neurons in neural networks often represent a mixture of unrelated
features. This phenomenon, called polysemanticity, can make interpreting neural
networks more difficult and so we aim to understand its causes. We propose
doing so through the lens of feature \emph{capacity}, which is the fractional
dimension each feature consumes in the embedding space. We show that in a toy
model the optimal capacity allocation tends to monosemantically represent the
most important features, polysemantically represent less important features (in
proportion to …

arxiv capacity networks neural networks