Technical Report: The Graph Spectral Token -- Enhancing Graph Transformers with Spectral Information

arXiv:2404.05604v1 Announce Type: new
Abstract: Graph Transformers have emerged as a powerful alternative to Message-Passing Graph Neural Networks (MP-GNNs) to address limitations such as over-squashing of information exchange. However, incorporating graph inductive bias into transformer architectures remains a significant challenge. In this report, we propose the Graph Spectral Token, a novel approach to directly encode graph spectral information, which captures the global structure of the graph, into the transformer architecture. By parameterizing the auxiliary [CLS] token and leaving other tokens …

abstract architectures arxiv bias challenge cs.lg gnns graph graph neural networks however inductive information limitations networks neural networks report technical the graph token transformer transformers type