Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation

arXiv:2406.16282v1 Announce Type: new
Abstract: Fine-tuning pretrained large models to downstream tasks is an important problem, which however suffers from huge memory overhead due to large-scale parameters. This work strives to reduce memory overhead in fine-tuning from perspectives of activation function and layer normalization. To this end, we propose the Approximate Backpropagation (Approx-BP) theory, which provides the theoretical feasibility of decoupling the forward and backward passes. We apply our Approx-BP theory to backpropagation training and derive memory-efficient alternatives of GELU …

arxiv backpropagation cs.ai cs.lg fine-tuning memory tuning type