Can We Use Large Language Models to Fill Relevance Judgment Holes?

arXiv:2405.05600v1 Announce Type: cross
Abstract: Incomplete relevance judgments limit the re-usability of test collections. When new systems are compared against previous systems used to build the pool of judged documents, they often do so at a disadvantage due to the ``holes'' in test collection (i.e., pockets of un-assessed documents returned by the new system). In this paper, we take initial steps towards extending existing test collections by employing Large Language Models (LLM) to fill the holes by leveraging and grounding …

abstract arxiv build collection cs.cl cs.ir documents judgment language language models large language large language models pool systems test type usability