SpaceByte: Towards Deleting Tokenization from Large Language Modeling

arXiv:2404.14408v1 Announce Type: cross
Abstract: Tokenization is widely used in large language models because it significantly improves performance. However, tokenization imposes several disadvantages, such as performance biases, increased adversarial vulnerability, decreased character-level modeling performance, and increased modeling complexity. To address these disadvantages without sacrificing performance, we propose SpaceByte, a novel byte-level decoder architecture that closes the performance gap between byte-level and subword autoregressive language modeling. SpaceByte consists of a byte-level Transformer model, but with extra larger transformer blocks inserted in …

abstract adversarial arxiv biases complexity cs.ai cs.cl cs.lg decoder disadvantages however language language models large language large language models modeling novel performance tokenization type vulnerability