Dissecting Language Models: Machine Unlearning via Selective Pruning

arXiv:2403.01267v1 Announce Type: new
Abstract: Understanding and shaping the behaviour of Large Language Models (LLMs) is increasingly important as applications become more powerful and more frequently adopted. This paper introduces a machine unlearning method specifically designed for LLMs. We introduce a selective pruning method for LLMs that removes neurons based on their relative importance on a targeted capability compared to overall network performance. This approach is a compute- and data-efficient method for identifying and removing neurons that enable specific behaviours. …

abstract applications arxiv become cs.cl cs.lg language language models large language large language models llms machine neurons paper pruning type understanding unlearning via