On the Fragility of Active Learners

arXiv:2403.15744v1 Announce Type: new
Abstract: Active learning (AL) techniques aim to maximally utilize a labeling budget by iteratively selecting instances that are most likely to improve prediction accuracy. However, their benefit compared to random sampling has not been consistent across various setups, e.g., different datasets, classifiers. In this empirical study, we examine how a combination of different factors might obscure any gains from an AL technique.
Focusing on text classification, we rigorously evaluate AL techniques over around 1000 experiments that …

abstract accuracy active learning aim arxiv benefit budget classifiers consistent cs.cl cs.lg datasets however instances labeling prediction random sampling study type