How Transformers Learn Causal Structure with Gradient Descent

arXiv:2402.14735v1 Announce Type: new
Abstract: The incredible success of transformers on sequence modeling tasks can be largely attributed to the self-attention mechanism, which allows information to be transferred between different parts of a sequence. Self-attention allows transformers to encode causal structure which makes them particularly suitable for sequence modeling. However, the process by which transformers learn such causal structure via gradient-based training algorithms remains poorly understood. To better understand this process, we introduce an in-context learning task that requires learning …

abstract arxiv attention cs.it cs.lg encode gradient information learn math.it modeling process self-attention stat.ml success tasks them transformers type