Faster Convergence for Transformer Fine-tuning with Line Search Methods

arXiv:2403.18506v1 Announce Type: new
Abstract: Recent works have shown that line search methods greatly increase performance of traditional stochastic gradient descent methods on a variety of datasets and architectures [1], [2]. In this work we succeed in extending line search methods to the novel and highly popular Transformer architecture and dataset domains in natural language processing. More specifically, we combine the Armijo line search with the Adam optimizer and extend it by subdividing the networks architecture into sensible units and …

abstract architecture architectures arxiv convergence cs.ai cs.lg dataset datasets faster fine-tuning gradient line novel performance popular search stochastic transformer transformer architecture type work