Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

arXiv:2402.19427v1 Announce Type: new
Abstract: Recurrent neural networks (RNNs) have fast inference and scale efficiently on long sequences, but they are difficult to train and hard to scale. We propose Hawk, an RNN with gated linear recurrences, and Griffin, a hybrid model that mixes gated linear recurrences with local attention. Hawk exceeds the reported performance of Mamba on downstream tasks, while Griffin matches the performance of Llama-2 despite being trained on over 6 times fewer tokens. We also show that …

abstract arxiv attention cs.cl cs.lg griffin hybrid inference language language models linear local attention networks neural networks recurrent neural networks rnn scale train type