Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

arXiv:2404.05868v1 Announce Type: new
Abstract: Large Language Models (LLMs) often memorize sensitive, private, or copyrighted data during pre-training. LLM unlearning aims to eliminate the influence of undesirable data from the pre-trained model while preserving the model's utilities on other tasks. Several practical methods have recently been proposed for LLM unlearning, mostly based on gradient ascent (GA) on the loss of undesirable data. However, on certain unlearning tasks, these methods either fail to effectively unlearn the target data or suffer from …

abstract arxiv cs.ai cs.cl cs.lg data influence language language models large language large language models llm llms negative optimization practical pre-trained model pre-training stat.ml tasks training type unlearning utilities