Information-Theoretic Distillation for Reference-less Summarization

arXiv:2403.13780v1 Announce Type: new
Abstract: The current winning recipe for automatic summarization is using proprietary large-scale language models (LLMs) such as ChatGPT as is, or imitation learning from them as teacher models. While increasingly ubiquitous dependence on such large-scale language models is convenient, there remains an important question of whether small-scale models could have achieved competitive results, if we were to seek an alternative learning method -- that allows for a more cost-efficient, controllable, yet powerful summarizer. We present InfoSumm, …

abstract arxiv chatgpt cs.ai cs.cl current distillation imitation learning information language language models llms proprietary question recipe reference scale small summarization them type