Learning Shortcuts: On the Misleading Promise of NLU in Language Models

The advent of large language models (LLMs) has enabled significant performance gains in the field of natural language processing. However, recent studies have found that LLMs often resort to shortcuts when performing tasks, creating an illusion of enhanced performance while lacking generalizability in their decision rules. This phenomenon introduces challenges in accurately assessing natural language understanding in LLMs. Our paper provides a concise survey of relevant research in this area and puts forth a perspective on the implications of shortcut …

cs.ai cs.cl decision found language language models language processing large language large language models llms natural natural language natural language processing nlu performance processing rules studies tasks