SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning

arXiv:2404.18239v1 Announce Type: new
Abstract: Large Language Models (LLMs) have highlighted the necessity of effective unlearning mechanisms to comply with data regulations and ethical AI practices. LLM unlearning aims at removing undesired data influences and associated model capabilities without compromising utility out of the scope of unlearning. While interest in studying LLM unlearning is growing,the impact of the optimizer choice for LLM unlearning remains under-explored. In this work, we shed light on the significance of optimizer selection in LLM unlearning …

abstract arxiv capabilities cs.cl cs.lg data data regulations ethical ethical ai language language models large language large language models llm llms optimization power practices regulations soul type unlearning utility while