[R] Unlimiformer: Long-Range Transformers with Unlimited Length Input

**Abstract**:

>Transformer-based models typically have a predefined bound to their input length, because of their need to potentially attend to every token in the input. In this work, we propose Unlimiformer: a general approach that can wrap any existing pretrained encoder-decoder transformer, and offload the attention computation across all layers to a single k-nearestneighbor index; this index can be kept on either the GPU or CPU memory and queried in sub-linear time. This way, we can index extremely long input …

abstract attention computation decoder encoder encoder-decoder general machinelearning transformer transformers work