Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps

arXiv:2302.00456v3 Announce Type: replace
Abstract: Transformers are ubiquitous in wide tasks. Interpreting their internals is a pivotal goal. Nevertheless, their particular components, feed-forward (FF) blocks, have typically been less analyzed despite their substantial parameter amounts. We analyze the input contextualization effects of FF blocks by rendering them in the attention maps as a human-friendly visualization scheme. Our experiments with both masked- and causal-language models reveal that FF networks modify the input contextualization to emphasize specific types of linguistic compositions. In …

abstract analyze arxiv attention components contextualization cs.cl effects lens maps pivotal rendering tasks them through transformers type