SEA: Sparse Linear Attention with Estimated Attention Mask

arXiv:2310.01777v2 Announce Type: replace-cross
Abstract: The transformer architecture has driven breakthroughs in recent years on tasks which require modeling pairwise relationships between sequential elements, as is the case in natural language understanding. However, long seqeuences pose a problem due to the quadratic complexity of the attention operation. Previous research has aimed to lower the complexity by sparsifying or linearly approximating the attention matrix. Yet, these approaches cannot straightforwardly distill knowledge from a teacher's attention matrix and often require complete retraining …

abstract architecture arxiv attention case complexity cs.cl cs.lg however language language understanding linear modeling natural natural language relationships research tasks transformer transformer architecture type understanding