Stronger Random Baselines for In-Context Learning

arXiv:2404.13020v1 Announce Type: cross
Abstract: Evaluating the in-context learning classification performance of language models poses challenges due to small dataset sizes, extensive prompt-selection using the validation set, and intentionally difficult tasks that lead to near-random performance. The standard random baseline -- the expected accuracy of guessing labels uniformly at random -- is stable when the evaluation set is used only once or when the dataset is large. We account for the common practice of validation set reuse and existing small …

abstract accuracy arxiv challenges classification context cs.cl cs.lg dataset in-context learning labels language language models near performance prompt random set small standard tasks type validation