ColA: Collaborative Adaptation with Gradient Learning

arXiv:2404.13844v1 Announce Type: new
Abstract: A primary function of back-propagation is to compute both the gradient of hidden representations and parameters for optimization with gradient descent. Training large models requires high computational costs due to their vast parameter sizes. While Parameter-Efficient Fine-Tuning (PEFT) methods aim to train smaller auxiliary models to save computational space, they still present computational overheads, especially in Fine-Tuning as a Service (FTaaS) for numerous users. We introduce Collaborative Adaptation (ColA) with Gradient Learning (GL), a parameter-free, …

abstract aim arxiv cola collaborative computational compute costs cs.ai cs.lg fine-tuning function gradient hidden large models optimization parameters peft propagation save space train training type vast