MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation

arXiv:2403.17698v1 Announce Type: new
Abstract: When the predicted sequence length exceeds the length seen during training, the transformer's inference accuracy diminishes. Existing relative position encoding methods, such as those based on the ALiBi technique, address the length extrapolation challenge exclusively through the implementation of a single kernel function, which introduces a constant bias to every post-softmax attention scores according to their distance. These approaches do not investigate or employ multiple kernel functions to address the extrapolation challenge. Drawing on the …

abstract accuracy arxiv challenge cs.ai cs.lg encoding function implementation inference kernel multiple positional encoding through training transformer type