LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements

arXiv:2404.06283v1 Announce Type: new
Abstract: The task of reading comprehension (RC), often implemented as context-based question answering (QA), provides a primary means to assess language models' natural language understanding (NLU) capabilities. Yet, when applied to large language models (LLMs) with extensive built-in world knowledge, this method can be deceptive. If the context aligns with the LLMs' internal knowledge, it is hard to discern whether the models' answers stem from context comprehension or from LLMs' internal information. Conversely, using data that …

abstract arxiv capabilities context cs.cl knowledge language language models language understanding large language large language models llms natural natural language nlu parametric question question answering reading type understanding world