Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmark

arXiv:2402.14359v1 Announce Type: new
Abstract: The summarization capabilities of pretrained and large language models (LLMs) have been widely validated in general areas, but their use in scientific corpus, which involves complex sentences and specialized knowledge, has been less assessed. This paper presents conceptual and experimental analyses of scientific summarization, highlighting the inadequacies of traditional evaluation methods, such as $n$-gram, embedding comparison, and QA, particularly in providing explanations, grasping scientific concepts, or identifying key content. Subsequently, we introduce the Facet-aware Metric …

abstract arxiv benchmark capabilities cs.cl evaluation experimental facet general highlighting knowledge language language models large language large language models llms metrics paper summarization type