How to use and interpret activation patching

arXiv:2404.15255v1 Announce Type: new
Abstract: Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results. We provide a summary of advice and best practices, based on our experience using this technique in practice. We include an overview of the different ways to apply activation patching and a discussion on how to interpret the results. We focus on what evidence patching experiments provide about circuits, and on …

abstract advice arxiv best practices cs.lg experience interpretability overview popular practice practices results summary type