Spectral Filters, Dark Signals, and Attention Sinks

arXiv:2402.09221v1 Announce Type: cross
Abstract: Projecting intermediate representations onto the vocabulary is an increasingly popular interpretation tool for transformer-based LLMs, also known as the logit lens. We propose a quantitative extension to this approach and define spectral filters on intermediate representations based on partitioning the singular vectors of the vocabulary embedding and unembedding matrices into bands. We find that the signals exchanged in the tail end of the spectrum are responsible for attention sinking (Xiao et al. 2023), of which …

abstract arxiv attention cs.ai cs.cl embedding extension filters intermediate interpretation llms partitioning popular quantitative singular tool transformer type vectors