Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding

arXiv:2405.02750v1 Announce Type: new
Abstract: Large language models (LLMs) tend to inadequately integrate input context during text generation, relying excessively on encoded prior knowledge in model parameters, potentially resulting in generated text with factual inconsistencies or contextually unfaithful content. LLMs utilize two primary knowledge sources: 1) prior (parametric) knowledge from pretraining, and 2) contextual (non-parametric) knowledge from input prompts. The study addresses the open question of how LLMs effectively balance these knowledge sources during the generation process, specifically in the …

arxiv cs.ai cs.cl decoding language language models large language large language models through type understanding