What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes

Polysemantic neurons -- neurons that activate for a set of unrelated features -- have been seen as a significant obstacle towards interpretability of task-optimized deep networks, with implications for AI safety. The classic origin story of polysemanticity is that the data contains more ``features" than neurons, such that learning to perform a task forces the network to co-allocate multiple unrelated features to the same neuron, endangering our ability to understand networks' internal processing. In this work, we present a second …

cs.ai cs.lg cs.ne data features interpretability mixed networks neurons safety set story