Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models

arXiv:2404.02936v1 Announce Type: cross
Abstract: The problem of pre-training data detection for large language models (LLMs) has received growing attention due to its implications in critical issues like copyright violation and test data contamination. The current state-of-the-art approach, Min-K%, measures the raw token probability which we argue may not be the most informative signal. Instead, we propose Min-K%++ to normalize the token probability with statistics of the categorical distribution over the whole vocabulary, which accurately reflects the relative likelihood of …

abstract art arxiv attention copyright cs.cl cs.lg current data detection language language models large language large language models llms pre-training probability raw state test token training training data type