Initialization is Critical to Whether Transformers Fit Composite Functions by Inference or Memorizing

arXiv:2405.05409v1 Announce Type: new
Abstract: Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate. In this work, we investigate the mechanisms of how transformers behave on unseen compositional tasks using anchor functions. We discover that the parameter initialization scale plays a critical role in determining whether the model learns inferential solutions, which capture the underlying compositional primitives, or symmetric solutions, which simply memorize mappings without understanding the compositional structure. By …

abstract anchor arxiv capabilities cs.lg functions inference performance tasks transformers type work