[Project] Unofficial implementation of Retentive Network (GitHub repo)

So very recently, a new paper was published to ArXiV called "Retentive Network: A Successor to Transformer for Large Language Models": [https://arxiv.org/abs/2307.08621](https://arxiv.org/abs/2307.08621). The title makes a fairly strong claim regarding the success of the model: transformers have long been established as among the best general-purpose learning techniques in the deep learning literature. Self-describing as a "successor to transformer" is therefore not to be taken lightly.

From what I can tell, the math checks out, and the authors demonstrate an intriguing …

attention authors checks core inference machinelearning math memory requirements retention transformer transformers