Improving Transformers using Faithful Positional Encoding

arXiv:2405.09061v1 Announce Type: new
Abstract: We propose a new positional encoding method for a neural network architecture called the Transformer. Unlike the standard sinusoidal positional encoding, our approach is based on solid mathematical grounds and has a guarantee of not losing information about the positional order of the input sequence. We show that the new encoding approach systematically improves the prediction performance in the time-series classification task.

abstract architecture arxiv cs.lg encoding improving information network network architecture neural network positional encoding show solid standard transformer transformers type