Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarization. (arXiv:2305.01951v3 [cs.CL] UPDATED)

Recent pre-trained language models (PLMs) achieve promising results in
existing abstractive summarization datasets. However, existing summarization
benchmarks overlap in time with the standard pre-training corpora and
finetuning datasets. Hence, the strong performance of PLMs may rely on the
parametric knowledge that is memorized during pre-training and fine-tuning.
Moreover, the knowledge memorized by PLMs may quickly become outdated, which
affects the generalization performance of PLMs on future data. In this work, we
propose TempoSum, a novel benchmark that contains data samples …

analysis arxiv benchmarks data datasets finetuning future knowledge language language models parametric performance pre-training standard summarization text text summarization training