Marathon: A Race Through the Realm of Long Context with Large Language Models

arXiv:2312.09542v2 Announce Type: replace
Abstract: With the advancement of large language models (LLMs) and the expansion of their context windows, existing long-context benchmarks fall short in effectively evaluating the models' comprehension and reasoning abilities in extended texts. Moreover, conventional benchmarks relying on F1 metrics often inaccurately score responses: they may undervalue correct answers that differ from the reference responses and overvalue incorrect ones that resemble the reference texts. In response to these limitations, we introduce Marathon, a novel evaluation benchmark …

arxiv context cs.cl language language models large language large language models marathon race realm replace through type