Looped Transformers are Better at Learning Learning Algorithms

arXiv:2311.12424v3 Announce Type: replace
Abstract: Transformers have demonstrated effectiveness in in-context solving data-fitting problems from various (latent) models, as reported by Garg et al. However, the absence of an inherent iterative structure in the transformer architecture presents a challenge in emulating the iterative algorithms, which are commonly employed in traditional machine learning methods. To address this, we propose the utilization of looped transformer architecture and its associated training methodology, with the aim of incorporating iterative characteristics into the transformer architectures. …

abstract algorithms architecture arxiv challenge context cs.lg cs.ne data however iterative machine machine learning traditional machine learning transformer transformer architecture transformers type