The Invalsi Benchmark: measuring Language Models Mathematical and Language understanding in Italian

arXiv:2403.18697v1 Announce Type: new
Abstract: While Italian is by all metrics a high resource language, currently, there are isn't a Language Model pre-trained exclusively in this language. This results in a lower number of available benchmarks to evaluate the performance of language models in Italian.
This work presents two new benchmarks to evaluate the models performance on mathematical understanding and language understanding in Italian. These benchmarks are based on real tests that are undertaken by students of age between 11 …

abstract arxiv benchmark benchmarks cs.cl isn italian language language model language models language understanding measuring metrics performance results type understanding work