Geotokens and Geotransformers

arXiv:2403.15940v1 Announce Type: new
Abstract: In transformer architectures, position encoding primarily provides a sense of sequence for input tokens. While the original transformer paper's method has shown satisfactory results in general language processing tasks, there have been new proposals, such as Rotary Position Embedding (RoPE), for further improvement. This paper presents geotokens, input components for transformers, each linked to a specific geological location. Unlike typical language sequences, for these tokens, the order is not as vital as the geographical coordinates …

abstract architectures arxiv components cs.ai cs.cl embedding encoding general improvement language language processing paper processing proposals results rope sense tasks tokens transformer transformers type