Locating and Editing Factual Associations in Mamba

arXiv:2404.03646v1 Announce Type: new
Abstract: We investigate the mechanisms of factual recall in the Mamba state space model. Our work is inspired by previous findings in autoregressive transformer language models suggesting that their knowledge recall is localized to particular modules at specific token locations; we therefore ask whether factual recall in Mamba can be similarly localized. To investigate this, we conduct four lines of experiments on Mamba. First, we apply causal tracing or interchange interventions to localize key components inside …

abstract arxiv cs.cl editing knowledge language language models locations mamba modules recall space state state space model token transformer transformer language models type work