Derivative-Free Optimization for Low-Rank Adaptation in Large Language Models

arXiv:2403.01754v1 Announce Type: new
Abstract: Parameter-efficient tuning methods such as LoRA could achieve comparable performance to model tuning by tuning a small portion of the parameters. However, substantial computational resources are still required, as this process involves calculating gradients and performing back-propagation throughout the model. Much effort has recently been devoted to utilizing the derivative-free optimization method to eschew the computation of gradients and showcase an augmented level of robustness in few-shot settings. In this paper, we prepend the low-rank …

abstract arxiv computational cs.cl free language language models large language large language models lora low low-rank adaptation optimization parameters performance process propagation resources small type