[R] RWKV-v2-RNN : A parallelizable RNN with transformer-level LM performance, and without using attention

Web: https://www.reddit.com/r/MachineLearning/comments/umq908/r_rwkvv2rnn_a_parallelizable_rnn_with/

Hi guys. I am an independent researcher and you might know me (BlinkDL) if you are in the EleutherAI discord.

I have built a RNN with transformer-level performance, without using attention. Moreover it supports both sequential & parallel mode in inference and training. So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx\_len, and free sentence embedding.

[https://github.com/BlinkDL/RWKV-LM](https://github.com/BlinkDL/RWKV-LM)

I am training a L24-D1024 RWKV-v2-RNN LM (430M params) on the Pile …

attention machinelearning performance rnn transformer