Language Modeling Is Compression

arXiv:2309.10668v2 Announce Type: replace
Abstract: It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has focused on training increasingly large and powerful self-supervised (language) models. Since these large language models exhibit impressive predictive capabilities, they are well-positioned to be strong compressors. In this work, we advocate for viewing the prediction problem through the lens of compression and evaluate the compression capabilities of large (foundation) …

abstract arxiv capabilities community compression cs.ai cs.cl cs.it cs.lg language language models large language large language models machine machine learning math.it modeling predictive predictive models training type