[N] (Update: Code Released) Landmark Attention: Random-Access Infinite Context Length for Transformers

Code for Landmark Attention is now released and it should be possible to finetune existing LLaMA models using this method.

[**https://github.com/epfml/landmark-attention**](https://github.com/epfml/landmark-attention)

* Paper: [https://arxiv.org/abs/2305.16300](https://arxiv.org/abs/2305.16300)

>The paper introduces a new method called Landmark Attention that addresses the memory limitations of transformers when dealing with longer contexts. The method allows access to the entire context while maintaining random-access flexibility, enabling the model to select any token in the context. It uses landmark tokens to represent blocks of input and trains the attention …

attention code context llama llama models machinelearning memory paper random transformers