How Much Annotation is Needed to Compare Summarization Models?

arXiv:2402.18756v1 Announce Type: new
Abstract: Modern instruction-tuned models have become highly capable in text generation tasks such as summarization, and are expected to be released at a steady pace. In practice one may now wish to choose confidently, but with minimal effort, the best performing summarization model when applied to a new domain or purpose. In this work, we empirically investigate the test sample size necessary to select a preferred model in the context of news summarization. Empirical results reveal …

abstract annotation arxiv become cs.cl instruction-tuned modern practice summarization tasks text text generation type