Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer

arXiv:2403.14552v1 Announce Type: new
Abstract: While Transformers have rapidly gained popularity in various computer vision applications, post-hoc explanations of their internal mechanisms remain largely unexplored. Vision Transformers extract visual information by representing image regions as transformed tokens and integrating them via attention weights. However, existing post-hoc explanation methods merely consider these attention weights, neglecting crucial information from the transformed tokens, which fails to accurately illustrate the rationales behind the models' predictions. To incorporate the influence of token transformation into interpretation, …

abstract applications arxiv attention computer computer vision cs.cv extract however image information them token tokens transformation transformer transformers type via vision vision transformers visual