all AI news
Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark. (arXiv:2304.11085v1 [cs.CL])
cs.CL updates on arXiv.org arxiv.org
Recent studies have demonstrated promising potential of ChatGPT for various
text annotation and classification tasks. However, ChatGPT is non-deterministic
which means that, as with human coders, identical input can lead to different
outputs. Given this, it seems appropriate to test the reliability of ChatGPT.
Therefore, this study investigates the consistency of ChatGPT's zero-shot
capabilities for text annotation and classification, focusing on different
model parameters, prompt variations, and repetitions of identical inputs. Based
on the real-world classification task of differentiating website …
annotation arxiv chatgpt classification coders human prompt reliability studies study test testing text website world