[R] SpaceByte: Towards Deleting Tokenization from Large Language Modeling - Rice University 2024 - Practically the same performance as subword tokenizers without their many downsides!

Paper: [https://arxiv.org/abs/2404.14408](https://arxiv.org/abs/2404.14408)

Github: [https://github.com/kjslag/spacebyte](https://github.com/kjslag/spacebyte)

Abstract:

>Tokenization is widely used in large language models because it significantly improves performance. However, **tokenization imposes several disadvantages, such as performance biases, increased adversarial vulnerability, decreased character-level modeling performance, and increased modeling complexity.** To address these disadvantages without sacrificing performance, we propose SpaceByte, a novel **byte-level decoder architecture that closes the performance gap between byte-level and subword autoregressive language modeling.** SpaceByte consists of a byte-level Transformer model, but with extra larger transformer blocks inserted in …

abstract machinelearning