DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models

arXiv:2309.03883v2 Announce Type: replace-cross
Abstract: Despite their impressive capabilities, large language models (LLMs) are prone to hallucinations, i.e., generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs that does not require conditioning on retrieved external knowledge nor additional fine-tuning. Our approach obtains the next-token distribution by contrasting the differences in logits obtained from projecting the later layers versus earlier layers to the vocabulary space, exploiting the fact that …

abstract arxiv capabilities cs.ai cs.cl cs.lg decoding facts hallucinations knowledge language language models large language large language models llms pretraining simple strategy type