Evaluation of Few-Shot Learning for Classification Tasks in the Polish Language

arXiv:2404.17832v1 Announce Type: new
Abstract: We introduce a few-shot benchmark consisting of 7 different classification tasks native to the Polish language. We conducted an empirical comparison with 0 and 16 shots between fine-tuning, linear probing, SetFit, and in-context learning (ICL) using various pre-trained commercial and open-source models. Our findings reveal that ICL achieves the best performance, with commercial models like GPT-3.5 and GPT-4 attaining the best performance. However, there remains a significant 14 percentage points gap between our best few-shot …

abstract arxiv benchmark classification commercial comparison context cs.cl evaluation few-shot few-shot learning fine-tuning in-context learning language linear open-source models setfit tasks type