Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variation

arXiv:2307.12973v2 Announce Type: replace
Abstract: Large Language Models (LLMs) exhibit remarkable text classification capabilities, excelling in zero- and few-shot learning (ZSL and FSL) scenarios. However, since they are trained on different datasets, performance varies widely across tasks between those models. Recent studies emphasize the importance of considering human label variation in data annotation. However, how this human label variation also applies to LLMs remains unexplored. Given this likely model specialization, we ask: Do aggregate LLM labels improve over individual models …

abstract arxiv capabilities classification cs.cl datasets few-shot few-shot learning however human importance instruction-tuned language language model language models large language large language models llms performance studies tasks text text classification type variation