Using Degeneracy in the Loss Landscape for Mechanistic Interpretability

arXiv:2405.10927v1 Announce Type: new
Abstract: Mechanistic Interpretability aims to reverse engineer the algorithms implemented by neural networks by studying their weights and activations. An obstacle to reverse engineering neural networks is that many of the parameters inside a network are not involved in the computation being implemented by the network. These degenerate parameters may obfuscate internal structure. Singular learning theory teaches us that neural network parameterizations are biased towards being more degenerate, and parameterizations with more degeneracy are likely to …

abstract algorithms arxiv computation cs.lg engineer engineering inside interpretability landscape loss network networks neural networks parameters studying type