HGRN2: Gated Linear RNNs with State Expansion

arXiv:2404.07904v1 Announce Type: new
Abstract: Hierarchically gated linear RNN (HGRN,Qin et al. 2023) has demonstrated competitive training speed and performance in language modeling, while offering efficient inference. However, the recurrent state size of HGRN remains relatively small, which limits its expressiveness.To address this issue, inspired by linear attention, we introduce a simple outer-product-based state expansion mechanism so that the recurrent state size can be significantly enlarged without introducing any additional parameters. The linear attention form also allows for hardware-efficient training.Our …

abstract arxiv attention cs.cl expansion however inference issue language linear modeling performance product rnn simple small speed state training type