Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting

arXiv:2306.17563v2 Announce Type: replace-cross
Abstract: Ranking documents using Large Language Models (LLMs) by directly feeding the query and candidate documents into the prompt is an interesting and practical problem. However, researchers have found it difficult to outperform fine-tuned baseline rankers on benchmark datasets. We analyze pointwise and listwise ranking prompts used by existing methods and argue that off-the-shelf LLMs do not fully understand these challenging ranking formulations. In this paper, we propose to significantly reduce the burden on LLMs by …

abstract analyze arxiv benchmark cs.cl cs.ir cs.lg datasets documents found however language language models large language large language models llms practical prompt prompting query ranking researchers text the prompt type