Gated Linear Attention Transformers with Hardware-Efficient Training

arXiv:2312.06635v4 Announce Type: replace
Abstract: Transformers with linear attention allow for efficient parallel training but can simultaneously be formulated as an RNN with 2D (matrix-valued) hidden states, thus enjoying linear-time inference complexity. However, linear attention generally underperforms ordinary softmax attention. Moreover, current implementations of linear attention lack I/O-awareness and are thus slower than highly optimized implementations of softmax attention. This work describes a hardware-efficient algorithm for linear attention that trades off memory movement against parallelizability. The resulting implementation, dubbed FLASHLINEARATTENTION, …

abstract arxiv attention complexity cs.cl cs.lg current hardware hidden inference linear matrix ordinary rnn softmax training transformers type