The Impact of LoRA on the Emergence of Clusters in Transformers

arXiv:2402.15415v1 Announce Type: new
Abstract: In this paper, we employ the mathematical framework on Transformers developed by \citet{sander2022sinkformers,geshkovski2023emergence,geshkovski2023mathematical} to explore how variations in attention parameters and initial token values impact the structural dynamics of token clusters. Our analysis demonstrates that while the clusters within a modified attention matrix dynamics can exhibit significant divergence from the original over extended periods, they maintain close similarities over shorter intervals, depending on the parameter differences. This work contributes to the fine-tuning field through practical …

abstract analysis arxiv attention cs.lg dynamics emergence explore framework impact lora math.ds matrix paper parameters stat.ml token transformers type values