Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons

arXiv:2405.05894v1 Announce Type: new
Abstract: LLM-as-a-judge approaches are a practical and effective way of assessing a range of text tasks, aligning with human judgements especially when applied in a comparative assessment fashion. However, when using pairwise comparisons to rank a set of candidates the computational costs scale quadratically with the number of candidates, which can have practical limitations. This paper introduces a Product of Expert (PoE) framework for efficient LLM Comparative Assessment. Here individual comparisons are considered experts that provide …

abstract arxiv assessment computational costs cs.cl experts fashion framework however human judge llm practical product scale set tasks text type