The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation

arXiv:2405.01299v1 Announce Type: new
Abstract: Large Language Models (LLMs) have emerged as powerful support tools across various natural language tasks and a range of application domains. Recent studies focus on exploring their capabilities for data annotation. This paper provides a comparative overview of twelve studies investigating the potential of LLMs in labelling data. While the models demonstrate promising cost and time-saving benefits, there exist considerable limitations, such as representativeness, bias, sensitivity to prompt variations and English language preference. Leveraging insights …

abstract analysis annotation application arxiv capabilities cs.ai cs.cl cs.lg data data annotation domains focus language language models large language large language models llms natural natural language overview paper representation studies support tasks tools type