Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance

arXiv:2403.00236v1 Announce Type: cross
Abstract: We investigate the performance of LLM-based zero-shot stance detection on tweets. Using FlanT5-XXL, an instruction-tuned open-source LLM, with the SemEval 2016 Tasks 6A, 6B, and P-Stance datasets, we study the performance and its variations under different prompts and decoding strategies, as well as the potential biases of the model. We show that the zero-shot approach can match or outperform state-of-the-art benchmarks, including fine-tuned models. We provide various insights into its performance including the sensitivity to …

abstract arxiv benchmarking cs.ai cs.cl cs.lg data datasets decoding detection insights instruction-tuned llm near performance prompting prompts sota strategies study tasks training training data tweets type zero-shot