Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models

arXiv:2310.07712v2 Announce Type: replace-cross
Abstract: Large language models (LLMs) exhibit positional bias in how they use context, which especially complicates listwise ranking. To address this, we propose permutation self-consistency, a form of self-consistency over ranking list outputs of black-box LLMs. Our key idea is to marginalize out different list orders in the prompt to produce an order-independent ranking with less positional bias. First, given some input prompt, we repeatedly shuffle the list in the prompt and pass it through the …

arxiv cs.cl cs.lg found language language models large language large language models ranking type