Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmark

arXiv:2405.01719v1 Announce Type: new
Abstract: We examine multi-task benchmarks in machine learning through the lens of social choice theory. We draw an analogy between benchmarks and electoral systems, where models are candidates and tasks are voters. This suggests a distinction between cardinal and ordinal benchmark systems. The former aggregate numerical scores into one model ranking; the latter aggregate rankings for each task. We apply Arrow's impossibility theorem to ordinal benchmarks to highlight the inherent limitations of ordinal systems, particularly their …

arxiv benchmark cs.lg diversity stability trade type