S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models

arXiv:2310.15147v2 Announce Type: replace
Abstract: The rapid development of Large Language Models (LLMs) has led to great strides in model capabilities like long-context understanding and reasoning. However, as LLMs are able to process longer contexts, it becomes more challenging to evaluate whether they have acquired certain capabilities, since the length of text (e.g., 200K tokens) they can process far exceeds what humans can reliably assess in a reasonable duration. In this paper, we propose using complex synthetic tasks as a …

abstract acquired arxiv capabilities context cs.cl development evaluation however language language models large language large language models llms process reasoning scalable synthetic type understanding