Length Extrapolation of Transformers: A Survey from the Perspective of Position Encoding

arXiv:2312.17044v3 Announce Type: replace
Abstract: Transformer has taken the field of natural language processing (NLP) by storm since its birth. Further, Large language models (LLMs) built upon it have captured worldwide attention due to its superior abilities. Nevertheless, all Transformer-based models including these powerful LLMs suffer from a preset length limit and can hardly generalize from short training sequences to longer inference ones, namely, they can not perform length extrapolation. Hence, a plethora of methods have been proposed to enhance …

abstract arxiv attention birth cs.cl encoding language language models language processing large language large language models llms natural natural language natural language processing nlp perspective processing storm survey transformer transformer-based models transformers type