Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questions

arXiv:2405.03205v1 Announce Type: cross
Abstract: Large Language Models (LLMs), such as the GPT-4 and LLaMA families, have demonstrated considerable success across diverse tasks, including multiple-choice questions (MCQs). However, these models exhibit a positional bias, particularly an even worse anchored bias in the GPT-2 family, where they consistently favour the first choice 'A' in MCQs during inference. This anchored bias challenges the integrity of GPT-2's decision-making process, as it skews performance based on the position rather than the content of the …

arxiv bias cs.ai cs.cl cs.lg gpt gpt-2 multiple questions type