Rethinking LLM Memorization through the Lens of Adversarial Compression

arXiv:2404.15146v1 Announce Type: new
Abstract: Large language models (LLMs) trained on web-scale datasets raise substantial concerns regarding permissible data usage. One major question is whether these models "memorize" all their training data or they integrate many data sources in some way more akin to how a human would learn and synthesize information. The answer hinges, to a large degree, on $\textit{how we define memorization}$. In this work, we propose the Adversarial Compression Ratio (ACR) as a metric for assessing memorization …

adversarial arxiv compression cs.cl cs.lg lens llm through type