Attention cannot be an Explanation. (arXiv:2201.11194v1 [cs.HC])

Web: http://arxiv.org/abs/2201.11194

Attention based explanations (viz. saliency maps), by providing
interpretability to black box models such as deep neural networks, are assumed
to improve human trust and reliance in the underlying models. Recently, it has
been shown that attention weights are frequently uncorrelated with
gradient-based measures of feature importance. Motivated by this, we ask a
follow-up question: "Assuming that we only consider the tasks where attention
weights correlate well with feature importance, how effective are these
attention based explanations in increasing human …

arxiv attention