Information Association for Language Model Updating by Mitigating LM-Logical Discrepancy

Large Language Models~(LLMs) struggle with providing current information due to the outdated pre-training data. Existing methods for updating LLMs, such as knowledge editing and continual fine-tuning, have significant drawbacks in generalizability of new information and the requirements on structured updating corpus. We identify the core challenge behind these drawbacks: the LM-logical discrepancy featuring the difference between language modeling probabilities and logical probabilities. To evaluate and address the core challenge, we propose a new task formulation of the information updating task …

association challenge continual core cs.cl current data editing fine-tuning identify information knowledge language language model language models large language large language models llms pre-training requirements struggle training training data