RULER: What's the Real Context Size of Your Long-Context Language Models?

arXiv:2404.06654v1 Announce Type: new
Abstract: The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands …

abstract arxiv context cs.cl form haystack however indicative information language language models lms retrieval simple test type