SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning

arXiv:2309.04766v3 Announce Type: replace
Abstract: We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we investigate the brittleness of foundation models in the dimensions of semantics and multilinguality. Our analyses span both open-sourced and closed models, leading to empirical results across classic NLP tasks, reasoning, and cultural comprehension. Key findings indicate …

abstract accuracy alignment arxiv benchmark cross-lingual cs.ai cs.cl foundation language metrics multilingual natural natural language practices reason reasoning standard type values