Interpreting Attention Layer Outputs with Sparse Autoencoders

arXiv:2406.17759v1 Announce Type: new
Abstract: Decomposing model activations into interpretable components is a key open problem in mechanistic interpretability. Sparse autoencoders (SAEs) are a popular method for decomposing the internal activations of trained transformers into sparse, interpretable features, and have been applied to MLP layers and the residual stream. In this work we train SAEs on attention layer outputs and show that also here SAEs find a sparse, interpretable decomposition. We demonstrate this on transformers from several model families and …

abstract arxiv attention autoencoders components cs.lg features interpretability key layer mlp popular problem residual stream transformers type work