Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers

arXiv:2302.01925v2 Announce Type: replace
Abstract: We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs). These include regular RPE techniques applied for sequential data, as well as novel RPEs operating on geometric data embedded in higher-dimensional Euclidean spaces. FLTs construct the optimal RPE mechanism implicitly by learning its spectral representation. As opposed to other architectures combining efficient low-rank linear attention with RPEs, FLTs remain practical in terms of …

abstract arxiv class cs.lg data embedded encoding fourier linear novel positional encoding transformers type