[R] Fine-Tuning Language Models with Just Forward Passes

This paper presents a memory-efficient zeroth-order optimizer (MeZO) for fine-tuning language models (LMs). As LMs grow larger, backpropagation becomes computationally costly, requiring large amounts of memory. MeZO adapts the classical Zeroth-order Stochastic Gradient Descent (ZO-SGD) method to operate in-place, enabling fine-tuning of LMs with the same memory footprint as inference.

For instance, with a single A100 80GB GPU, MeZO can train a 30-billion parameter model, whereas fine-tuning with backpropagation can only train a 2.7-billion parameter LM with the same resources. …

backpropagation enabling fine-tuning gradient inference language language models machinelearning memory paper stochastic