Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand. (arXiv:2112.04139v2 [cs.CL] UPDATED)

Natural language processing researchers have identified limitations of
evaluation methodology for generation tasks, with new questions raised about
the validity of automatic metrics and of crowdworker judgments. Meanwhile,
efforts to improve generation models tend to depend on simple n-gram overlap
metrics (e.g., BLEU, ROUGE). We argue that new advances on models and metrics
should each more directly benefit and inform the other. We therefore propose a
generalization of leaderboards, bidimensional leaderboards (Billboards), that
simultaneously tracks progress in language generation models …

arxiv language