[R] Formal Algorithms for Transformers (DeepMind, 2022)

Web: https://www.reddit.com/r/MachineLearning/comments/wh1q99/r_formal_algorithms_for_transformers_deepmind_2022/

Paper: [https://arxiv.org/abs/2207.09238](https://arxiv.org/abs/2207.09238?fbclid=IwAR0Qc3SGyOu9zg34hRMLExUPNytAPkQrlZP4G8yhXPg11_EW3MuZtPKeWnM)

Abstract:

>This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (\*not\* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.

​

https://preview.redd.it/h53zcmn4nxf91.jpg?width=596&format=pjpg&auto=webp&s=86bb06604f6987379392d97324357f2ea5b19ac2

algorithms deepmind machinelearning transformers