MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks

arXiv:2311.07463v2 Announce Type: replace
Abstract: There has been a surge in LLM evaluation research to understand LLM capabilities and limitations. However, much of this research has been confined to English, leaving LLM building and evaluation for non-English languages relatively unexplored. Several new LLMs have been introduced recently, necessitating their evaluation on non-English languages. This study aims to perform a thorough evaluation of the non-English capabilities of SoTA LLMs (GPT-3.5-Turbo, GPT-4, PaLM2, Gemini-Pro, Mistral, Llama2, and Gemma) by comparing them on …

abstract arxiv benchmarking building capabilities cs.cl english evaluation however language language models languages large language large language models limitations llm llms research tasks type