Fact or Fiction? NOCHA: A New Benchmark for Evaluating Long-Context Reasoning in LLMs

Researchers from UMass Amherst, Allen Institute for AI, and Princeton University have introduced a new evaluation methodology called NOCHA (Narrative Open-Contextualized Human Annotation). This approach is designed to assess the performance of long-context language models more accurately. NOCHA involves collecting minimal narrative pairs, where one claim is true, and the other is false, both written by readers of books.

The NOCHA methodology involves collecting narrative minimal pairs from recently published fictional books. Annotators familiar with these books generate pairs of …

allen allen institute allen institute for ai annotation benchmark claim context evaluation fiction human institute language language models llms machinelearningnews methodology narrative performance princeton university reasoning researchers university