Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment

arXiv:2311.08596v2 Announce Type: replace
Abstract: The interactive nature of Large Language Models (LLMs) theoretically allows models to refine and improve their answers, yet systematic analysis of the multi-turn behavior of LLMs remains limited. In this paper, we propose the FlipFlop experiment: in the first round of the conversation, an LLM completes a classification task. In a second round, the LLM is challenged with a follow-up phrase like "Are you sure?", offering an opportunity for the model to reflect on its …

abstract analysis arxiv behavior cs.cl experiment interactive language language models large language large language models leads llms nature paper performance refine type