[R] RWKV-4 7B release: an attention-free RNN language model matching GPT-J performance (14B training in progress)

Hi everyone. I have finished training RWKV-4 7B (an attention-free RNN LLM) and it can match GPT-J (6B params) performance. **Maybe RNN is already all you need** :)

https://preview.redd.it/71cce2y75j0a1.png?width=1336&format=png&auto=webp&s=5af76abc4f42fd63f0194ee93f78db01c1b21d97

Previous discussion: [https://www.reddit.com/r/MachineLearning/comments/xfup9f/r\_rwkv4\_scaling\_rnn\_to\_7b\_params\_and\_beyond\_with/](https://www.reddit.com/r/MachineLearning/comments/xfup9f/r_rwkv4_scaling_rnn_to_7b_params_and_beyond_with/)

RWKV has both RNN & GPT mode. The RNN mode is great for inference. The GPT mode is great for training. Both modes are faster than usual transformer and saves VRAM, because the self-attention mechanism is replaced by simpler (almost linear) formulas. Moreover the hidden state is tiny …

attention free gpt gpt-j language language model machinelearning performance progress release rnn training