Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense

arXiv:2405.04655v1 Announce Type: new
Abstract: Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. However, their understanding of cultural commonsense remains largely unexamined. In this paper, we conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. Using several general and cultural commonsense benchmarks, we find that (1) LLMs have a significant discrepancy in performance when tested on culture-specific commonsense knowledge for different cultures; (2) LLMs' …

abstract art arxiv benchmark capabilities commonsense context cs.cl however language language models large language large language models limitations llms paper state through type understanding