Unfamiliar Finetuning Examples Control How Language Models Hallucinate

arXiv:2403.05612v1 Announce Type: new
Abstract: Large language models (LLMs) have a tendency to generate plausible-sounding yet factually incorrect responses, especially when queried on unfamiliar concepts. In this work, we explore the underlying mechanisms that govern how finetuned LLMs hallucinate. Our investigation reveals an interesting pattern: as inputs become more unfamiliar, LLM outputs tend to default towards a ``hedged'' prediction, whose form is determined by how the unfamiliar examples in the finetuning data are supervised. Thus, by strategically modifying these examples' …

abstract arxiv become concepts control cs.ai cs.cl cs.lg examples explore finetuning generate inputs investigation language language models large language large language models llm llms responses type work