Simulation of Graph Algorithms with Looped Transformers

The execution of graph algorithms using neural networks has recently attracted significant interest due to promising empirical progress. This motivates further understanding of how neural networks can replicate reasoning steps with relational data. In this work, we study the ability of transformer networks to simulate algorithms on graphs from a theoretical perspective. The architecture that we utilize is a looped transformer with extra attention heads that interact with the graph. We prove by construction that this architecture can simulate algorithms …

algorithms cs.ai cs.ds cs.lg data graph graph algorithms graphs networks neural networks perspective progress reasoning relational replicate simulation study transformer transformers understanding work