Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?

arXiv:2402.12483v1 Announce Type: new
Abstract: Multiple-choice question answering (MCQA) is often used to evaluate large language models (LLMs). To see if MCQA assesses LLMs as intended, we probe if LLMs can perform MCQA with choices-only prompts, where models must select the correct answer only from the choices. In three MCQA datasets and four LLMs, this prompt bests a majority baseline in 11/12 cases, with up to 0.33 accuracy gain. To help explain this behavior, we conduct an in-depth, black-box analysis …

abduction abstract arxiv cs.cl language language models large language large language models llms multiple probe prompts question question answering questions type