Pragmatic Competence Evaluation of Large Language Models for Korean

arXiv:2403.12675v1 Announce Type: new
Abstract: The current evaluation of Large Language Models (LLMs) predominantly relies on benchmarks focusing on their embedded knowledge by testing through multiple-choice questions (MCQs), a format inherently suited for automated evaluation. Our study extends this evaluation to explore LLMs' pragmatic competence--a facet previously underexamined before the advent of sophisticated LLMs, specifically in the context of Korean. We employ two distinct evaluation setups: the conventional MCQ format, adapted for automatic evaluation, and Open-Ended Questions (OEQs), assessed by …

abstract arxiv automated benchmarks cs.cl current embedded evaluation explore facet format knowledge language language models large language large language models llms multiple questions study testing through type