The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Tasks

In the realm of Computational Social Science (CSS), practitioners often navigate complex, low-resource domains and face the costly and time-intensive challenges of acquiring and annotating data. We aim to establish a set of guidelines to address such challenges, comparing the use of human-labeled data with synthetically generated data from GPT-4 and Llama-2 in ten distinct CSS classification tasks of varying complexity. Additionally, we examine the impact of training data sizes on performance. Our findings reveal that models trained on human-labeled …

aim augmented data challenges classification computational cs.cl cs.cy css data domains face generated gpt gpt-4 guidelines human llm low parrot physics.soc-ph science set social social science tasks