Functional Interpolation for Relative Positions Improves Long Context Transformers

arXiv:2310.04418v2 Announce Type: replace
Abstract: Preventing the performance decay of Transformers on inputs longer than those used for training has been an important challenge in extending the context length of these models. Though the Transformer architecture has fundamentally no limits on the input sequence lengths it can process, the choice of position encoding used during training can limit the performance of these models on longer inputs. We propose a novel functional relative position encoding with progressive interpolation, FIRE, to improve …

abstract architecture arxiv challenge context cs.lg functional inputs performance process training transformer transformer architecture transformers type