Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks

arXiv:2406.13469v1 Announce Type: new
Abstract: This paper explores the performance of encoder and decoder language models on multilingual Natural Language Understanding (NLU) tasks, with a broad focus on Germanic languages. Building upon the ScandEval benchmark, which initially was restricted to evaluating encoder models, we extend the evaluation framework to include decoder models. We introduce a method for evaluating decoder models on NLU tasks and apply it to the languages Danish, Swedish, Norwegian, Icelandic, Faroese, German, Dutch, and English. Through a …

abstract analysis arxiv benchmark building comparative analysis cs.ai cs.cl cs.lg decoder encoder focus language language models languages language understanding multilingual natural natural language nlu paper performance tasks type understanding