On the Convergence of Zeroth-Order Federated Tuning in Large Language Models

The confluence of Federated Learning (FL) and Large Language Models (LLMs) is ushering in a new era in privacy-preserving natural language processing. However, the intensive memory requirements for fine-tuning LLMs pose significant challenges, especially when deploying on edge devices with limited computational resources. To circumvent this, we explore the novel integration of Memory-efficient Zeroth-Order Optimization within a federated setting, a synergy we denote as FedMeZO. Our study is the first to examine the theoretical underpinnings of FedMeZO in the context …

challenges computational confluence convergence cs.cl cs.lg devices edge edge devices explore federated learning fine-tuning language language models language processing large language large language models llms memory natural natural language natural language processing privacy processing requirements resources