MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification. (arXiv:2205.12394v1 [cs.CL])

In text summarization and simplification, system outputs must be evaluated
along multiple dimensions such as relevance, factual consistency, fluency, and
grammaticality, and a wide range of possible outputs could be of high quality.
These properties make the development of an adaptable, reference-less
evaluation metric both necessary and challenging. We introduce MaskEval, a
reference-less metric for text summarization and simplification that operates
by performing masked language modeling (MLM) on the concatenation of the
candidate and the source texts. It features an …

arxiv evaluation summarization text text summarization