Infinite Limits of Multi-head Transformer Dynamics

arXiv:2405.15712v1 Announce Type: cross
Abstract: In this work, we analyze various scaling limits of the training dynamics of transformer models in the feature learning regime. We identify the set of parameterizations that admit well-defined infinite width and depth limits, allowing the attention layers to update throughout training--a relevant notion of feature learning in these models. We then use tools from dynamical mean field theory (DMFT) to analyze various infinite limits (infinite key/query dimension, infinite heads, and infinite depth) which have …

abstract analyze arxiv attention cond-mat.dis-nn cs.lg dynamics feature head identify multi-head notion scaling set stat.ml training transformer transformer models type update work