DPZero: Private Fine-Tuning of Language Models without Backpropagation

arXiv:2310.09639v2 Announce Type: replace
Abstract: The widespread practice of fine-tuning large language models (LLMs) on domain-specific data faces two major challenges in memory and privacy. First, as the size of LLMs continues to grow, the memory demands of gradient-based training methods via backpropagation become prohibitively high. Second, given the tendency of LLMs to memorize training data, it is important to protect potentially sensitive information in the fine-tuning data from being regurgitated. Zeroth-order methods, which rely solely on forward passes, substantially …

abstract arxiv backpropagation become challenges cs.cr cs.lg data domain fine-tuning gradient language language models large language large language models llms major math.oc memory practice privacy stat.ml training type via