Simplifying Transformer Blocks

arXiv:2311.01906v2 Announce Type: replace
Abstract: A simple design recipe for deep Transformers is to compose identical building blocks. But standard transformer blocks are far from simple, interweaving attention and MLP sub-blocks with skip connections & normalisation layers in precise arrangements. This complexity leads to brittle architectures, where seemingly minor changes can significantly reduce training speed, or render models untrainable.
In this work, we ask to what extent the standard transformer block can be simplified? Combining signal propagation theory and empirical …

abstract architectures arxiv attention building complexity compose cs.lg design leads mlp recipe reduce render replace simple simplifying speed standard training transformer transformers type