Brainformers: Trading Simplicity for Efficiency

arXiv:2306.00008v2 Announce Type: replace
Abstract: Transformers are central to recent successes in natural language processing and computer vision. Transformers have a mostly uniform backbone where layers alternate between feed-forward and self-attention in order to build a deep network. Here we investigate this design choice and find that more complex blocks that have different permutations of layer primitives can be more efficient. Using this insight, we develop a complex block, named Brainformer, that consists of a diverse sets of layers such …

abstract arxiv attention build computer computer vision cs.cl cs.lg design efficiency language language processing natural natural language natural language processing network permutations processing self-attention simplicity trading transformers type uniform vision