HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization

arXiv:2402.16694v1 Announce Type: new
Abstract: Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs. In response, we introduce HumanEval-XL, a massively multilingual code generation benchmark specifically crafted to …

arxiv benchmark code code generation cross-lingual cs.cl cs.pl cs.se humaneval language multilingual natural natural language type