Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning

arXiv:2402.15751v1 Announce Type: new
Abstract: While fine-tuning large language models (LLMs) for specific tasks often yields impressive results, it comes at the cost of memory inefficiency due to back-propagation in gradient-based training. Memory-efficient Zeroth-order (MeZO) optimizers, recently proposed to address this issue, only require forward passes during training, making them more memory-friendly. However, the quality of gradient estimates in zeroth order optimization often depends on the data dimensionality, potentially explaining why MeZO still exhibits significant performance drops compared to standard …

abstract arxiv cost cs.ai cs.cl cs.lg fine-tuning gradient issue language language models large language large language models llm llms making memory parameters performance propagation results specific tasks tasks training type