[R] HGRN2: Gated Linear RNNs with State Expansion

**Paper**: [https://arxiv.org/abs/2404.07904](https://arxiv.org/abs/2404.07904)

**Code**: [https://github.com/OpenNLPLab/HGRN2](https://github.com/OpenNLPLab/HGRN2)

**Standalone code** (1): [https://github.com/Doraemonzzz/hgru2-pytorch](https://github.com/Doraemonzzz/hgru2-pytorch)

**Standalone code** (2): [https://github.com/sustcsonglin/flash-linear-attention/tree/main/fla/models/hgrn2](https://github.com/sustcsonglin/flash-linear-attention/tree/main/fla/models/hgrn2)

**Abstract**:

>**Hierarchically gated linear RNN** (**HGRN**, Qin et al. 2023) has demonstrated competitive training speed and performance in language modeling, while offering efficient inference. However, the recurrent state size of HGRN remains relatively small, which limits its expressiveness. To address this issue, inspired by linear attention, we introduce a simple outer-product-based state expansion mechanism so that the recurrent state size can be significantly enlarged without introducing any additional …

abstract attention expansion however inference issue language linear machinelearning modeling performance product rnn simple small speed state training while