[R] Recurrent Memory has broken the limits of Context Length for Transformer Neural Networks

The researchers segmented the sequence and added special memory tokens to the input: memory states from the output of the previous segment became inputs for the next one. Thus, a whole transformer acts as a recurrent cell, and memory serves as the recurrent state of the network. This approach was called Recurrent Memory Transformer (RMT).

The authors augmented small transformer models like BERT and GPT-2 with this memory and tested them on various question-answering tasks where facts needed for answering …

context inputs machinelearning memory networks neural networks next researchers segment state tokens transformer