Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankers

arXiv:2404.03192v1 Announce Type: cross
Abstract: The integration of Large Language Models (LLMs) in information retrieval has raised a critical reevaluation of fairness in the text-ranking models. LLMs, such as GPT models and Llama2, have shown effectiveness in natural language understanding tasks, and prior works (e.g., RankGPT) have also demonstrated that the LLMs exhibit better performance than the traditional ranking models in the ranking task. However, their fairness remains largely unexplored. This paper presents an empirical study evaluating these LLMs using …

abstract arxiv cs.cl cs.ir fairness gpt gpt models information integration language language models language understanding large language large language models llama2 llms natural natural language prior ranking retrieval study tasks text type understanding