[P] Spin-Model Transformers

A non-equilibrium statistical mechanics perspective on transformers.

We present a class of transformers based on mean-field dynamics of vector-spin models. Our framework supports asymmetric couplings and yields residual, attention, and feed-forward terms.

Post: https://mcbal.github.io/post/spin-model-transformers

Code (JAX): https://github.com/mcbal/spin-model-transformers

attention dynamics equilibrium framework machinelearning mean perspective residual spin statistical terms transformers vector