Training LLMs over Neurally Compressed Text

arXiv:2404.03626v1 Announce Type: cross
Abstract: In this paper, we explore the idea of training large language models (LLMs) over highly compressed text. While standard subword tokenizers compress text by a small factor, neural text compressors can achieve much higher rates of compression. If it were possible to train LLMs directly over neurally compressed text, this would confer advantages in training and serving efficiency, as well as easier handling of long text spans. The main obstacle to this goal is that …

abstract arxiv compression cs.cl cs.lg explore language language models large language large language models llms paper small standard text train training training llms type