When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models

arXiv:2404.09129v1 Announce Type: new
Abstract: Recent studies suggest that self-reflective prompting can significantly enhance the reasoning capabilities of Large Language Models (LLMs). However, the use of external feedback as a stop criterion raises doubts about the true extent of LLMs' ability to emulate human-like self-reflection. In this paper, we set out to clarify these capabilities under a more stringent evaluation setting in which we disallow any kind of external feedback. Our findings under this setting show a split: while self-reflection …

arxiv cs.cl language language models large language large language models testing thinking type