Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models

arXiv:2404.09338v1 Announce Type: new
Abstract: Large language models (LLMs) exhibit impressive natural language capabilities but suffer from hallucination -- generating content ungrounded in the realities of training data. Recent work has focused on decoding techniques to improve factuality during inference by leveraging LLMs' hierarchical representation of factual knowledge, manipulating the predicted distributions at inference time. Current state-of-the-art approaches refine decoding by contrasting early-exit distributions from a lower layer with the final layer to exploit information related to factuality within the …

abstract arxiv capabilities cs.cl data decoding entropy hallucination hierarchical inference knowledge language language models large language large language models llms natural natural language representation training training data type work