Understanding Finetuning for Factual Knowledge Extraction

arXiv:2406.14785v1 Announce Type: new
Abstract: In this work, we study the impact of QA fine-tuning data on downstream factuality. We show that fine-tuning on lesser-known facts that are poorly stored during pretraining yields significantly worse factuality than fine-tuning on well-known facts, even when all facts are seen during pretraining. We prove this phenomenon theoretically, showing that training on lesser-known facts can lead the model to ignore subject entity names and instead output a generic plausible response even when the relevant …

abstract arxiv cs.cl cs.lg data extraction facts fine-tuning finetuning impact knowledge pretraining prove show study tuning type understanding work