XL$^2$Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies

arXiv:2404.05446v1 Announce Type: new
Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks but are constrained by their small context window sizes. Various efforts have been proposed to expand the context window to accommodate even up to 200K input tokens. Meanwhile, building high-quality benchmarks with much longer text lengths and more demanding tasks to provide comprehensive evaluations is of immense practical interest to facilitate long context understanding research of LLMs. However, prior benchmarks create datasets that ostensibly …

abstract arxiv benchmark benchmarks building context context window cs.cl dependencies diverse expand language language models large language large language models llms performance quality small tasks tokens type understanding