Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?

arXiv:2405.05904v1 Announce Type: new
Abstract: When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training. It is often conjectured that this can teach the model the behavior of hallucinating factually incorrect responses, as the model is trained to generate facts that are not grounded in its pre-existing knowledge. In this work, we study the impact of such exposure to new knowledge on the capability of the fine-tuned model to …

abstract acquired arxiv behavior cs.cl facts fine-tuning generate hallucinations information knowledge language language models large language large language models llms pre-training responses supervised fine-tuning through training type via