Linearizing Large Language Models

arxiv: [https://arxiv.org/abs/2405.06640](https://arxiv.org/abs/2405.06640)

tweet: [https://x.com/sedrickkeh2/status/1790756596102074652](https://x.com/sedrickkeh2/status/1790756596102074652)

Transform Mistral into an RNN in a scalable way. Beats RWKV with <10% of the data.

data deeplearning mistral rnn rwkv scalable