RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian

arXiv:2402.13222v1 Announce Type: new
Abstract: Recently, large language models (LLMs) have become increasingly powerful and have become capable of solving a plethora of tasks through proper instructions in natural language. However, the vast majority of testing suites assume that the instructions are written in English, the de facto prompting language. Code intelligence and problem solving still remain a difficult task, even for the most advanced LLMs. Currently, there are no datasets to measure the generalization power for code-generation models in …

abstract arxiv become code code intelligence cs.cl dataset definitions english intelligence language language models large language large language models llms measuring natural natural language tasks testing through type vast