Why "classic" Transformers are shallow and how to make them go deep

Since its introduction in 2017, Transformer has emerged as the leading neural network architecture, catalyzing revolutionary advancements in many AI disciplines. The key innovation in Transformer is a Self-Attention (SA) mechanism designed to capture contextual information. However, extending the original Transformer design to models of greater depth has proven exceedingly challenging, if not impossible. Even though various modifications have been proposed in order to stack more layers of SA mechanism into deeper models, a full understanding of this depth problem …

architecture attention cs.ai cs.lg cs.ne design information innovation introduction key network network architecture neural network self-attention the key them transformer transformers