Quantifying Memorization of Domain-Specific Pre-trained Language Models using Japanese Newspaper and Paywalls

arXiv:2404.17143v1 Announce Type: new
Abstract: Dominant pre-trained language models (PLMs) have been successful in high-quality natural language generation. However, the analysis of their generation is not mature: do they acquire generalizable linguistic abstractions, or do they simply memorize and recover substrings of the training data? Especially, few studies focus on domain-specific PLM. In this study, we pre-trained domain-specific GPT-2 models using a limited corpus of Japanese newspaper articles and quantified memorization of training data by comparing them with general Japanese …

abstract abstractions analysis arxiv cs.cl data domain however japanese language language generation language models natural natural language natural language generation quality studies training training data type