Length Generalization of Causal Transformers without Position Encoding

arXiv:2404.12224v1 Announce Type: new
Abstract: Generalizing to longer sentences is important for recent Transformer-based language models. Besides algorithms manipulating explicit position features, the success of Transformers without position encodings (NoPE) provides a new way to overcome the challenge. In this paper, we study the length generalization property of NoPE. We find that although NoPE can extend to longer sequences than the commonly used explicit position encodings, it still has a limited context length. We identify a connection between the failure …

abstract algorithms arxiv causal challenge cs.cl encoding features language language models paper property study success transformer transformers type