PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models

arXiv:2405.04585v1 Announce Type: cross
Abstract: There are several improvements proposed over the baseline Absolute Positional Encoding (APE) method used in original transformer. In this study, we aim to investigate the implications of inadequately representing positional encoding in higher dimensions on crucial aspects of the attention mechanism, the model's capacity to learn relative positional information, and the convergence of models, all stemming from the choice of sinusoidal basis functions. Through a combination of theoretical insights and empirical analyses, we elucidate how …

abstract aim ape arxiv attention capacity cs.ai cs.cl cs.lg dimensions encoding improvements language language models large language large language models pope positional encoding study transformer type