Eight Methods to Evaluate Robust Unlearning in LLMs

arXiv:2402.16835v1 Announce Type: new
Abstract: Machine unlearning can be useful for removing harmful capabilities and memorized text from large language models (LLMs), but there are not yet standardized methods for rigorously evaluating it. In this paper, we first survey techniques and limitations of existing unlearning evaluations. Second, we apply a comprehensive set of tests for the robustness and competitiveness of unlearning in the "Who's Harry Potter" (WHP) model from Eldan and Russinovich (2023). While WHP's unlearning generalizes well when evaluated …

abstract apply arxiv capabilities cs.cl language language models large language large language models limitations llms machine paper robust set survey text type unlearning