Shifting order in multiple-choice questions massively affects LLM performance

Recent research proposes that Large Language Models (LLMs) may not be as reliable as we think. In fact, the order of options in a multiple-choice question drastically influences the responses from LLMs such as GPT-4 and InstructGPT.

If you want to stay on top of the latest trends and insights in AI and tech, [look here first.](https://supercharged-ai.beehiiv.com/subscribe?utm_source=reddit&utm_medium=llm-performance&utm_campaign=campaign)

https://preview.redd.it/dxfsq72kzalb1.png?width=1289&format=png&auto=webp&s=e4ed5b541073bde18d2865f2c15e8028388070f5

**What are the findings?**

* **LLM sensitivity to multiple-choice arrangement:** The study suggests if options in multiple-choice questions are reordered, the LLM's …

artificial gpt gpt-4 instructgpt language language models large language large language models llm llm performance llms multiple performance questions research responses sensitivity study think