MambaByte: Token-free Selective State Space Model

arXiv:2401.13660v2 Announce Type: replace-cross
Abstract: Token-free language models learn directly from raw bytes and remove the inductive bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences. In this setting, standard autoregressive Transformers scale poorly as the effective memory required grows with sequence length. The recent development of the Mamba state space model (SSM) offers an appealing alternative approach with a fixed-sized memory state and efficient decoding. We propose MambaByte, a token-free adaptation of the Mamba SSM …

abstract arxiv autoregressive transformers bias cs.cl cs.lg development free however inductive language language models learn mamba memory raw results scale space standard state state space model token tokenization transformers type