Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models

arXiv:2404.06209v1 Announce Type: new
Abstract: While many have shown how Large Language Models (LLMs) can be applied to a diverse set of tasks, the critical issues of data contamination and memorization are often glossed over. In this work, we address this concern for tabular data. Specifically, we introduce a variety of different techniques to assess whether a language model has seen a tabular dataset during training. This investigation reveals that LLMs have memorized many popular tabular datasets verbatim. We then …

arxiv cs.ai cs.cl cs.lg data elephants language language models large language large language models tabular tabular data type