A Meta-Learning Perspective on Transformers for Causal Language Modeling

arXiv:2310.05884v2 Announce Type: replace
Abstract: The Transformer architecture has become prominent in developing large causal language models. However, mechanisms to explain its capabilities are not well understood. Focused on the training process, here we establish a meta-learning view of the Transformer architecture when trained for the causal language modeling task, by explicating an inner optimization process within the Transformer. Further, within the inner optimization, we discover and theoretically analyze a special characteristic of the norms of learned token representations within …

abstract architecture arxiv become capabilities causal cs.ai cs.cl cs.lg however language language models meta meta-learning modeling perspective process training transformer transformer architecture transformers type view