From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency

arXiv:2404.12145v1 Announce Type: new
Abstract: The staggering pace with which the capabilities of large language models (LLMs) are increasing, as measured by a range of commonly used natural language understanding (NLU) benchmarks, raises many questions regarding what "understanding" means for a language model and how it compares to human understanding. This is especially true since many LLMs are exclusively trained on text, casting doubt on whether their stellar benchmark performances are reflective of a true understanding of the problems represented …

abstract arxiv benchmarks capabilities cs.ai cs.cl form language language model language models language understanding large language large language models llms meaning natural natural language nlu questions raises semantic type understanding