What's under the hood: Investigating Automatic Metrics on Meeting Summarization

arXiv:2404.11124v1 Announce Type: new
Abstract: Meeting summarization has become a critical task considering the increase in online interactions. While new techniques are introduced regularly, their evaluation uses metrics not designed to capture meeting-specific errors, undermining effective evaluation. This paper investigates what the frequently used automatic metrics capture and which errors they mask by correlating automatic metric scores with human evaluations across a broad error taxonomy. We commence with a comprehensive literature review on English meeting summarization to define key challenges …

abstract arxiv become cs.ai cs.cl errors evaluation interactions metrics paper summarization type