Return of EM: Entity-driven Answer Set Expansion for QA Evaluation

arXiv:2404.15650v1 Announce Type: new
Abstract: Recently, directly using large language models (LLMs) has been shown to be the most reliable method to evaluate QA models. However, it suffers from limited interpretability, high cost, and environmental harm. To address these, we propose to use soft EM with entity-driven answer set expansion. Our approach expands the gold answer set to include diverse surface forms, based on the observation that the surface forms often follow particular patterns depending on the entity type. The …

abstract arxiv cost cs.cl environmental evaluation expansion harm however interpretability language language models large language large language models llms set type