Zero-Shot Tokenizer Transfer

arXiv:2405.07883v1 Announce Type: new
Abstract: Language models (LMs) are bound to their tokenizer, which maps raw text to a sequence of vocabulary items (tokens). This restricts their flexibility: for example, LMs trained primarily on English may still perform well in other natural and programming languages, but have vastly decreased efficiency due to their English-centric tokenizer. To mitigate this, we should be able to swap the original LM tokenizer with an arbitrary one, on the fly, without degrading performance. Hence, in …

abstract arxiv cs.cl efficiency english example flexibility language language models languages lms maps natural programming programming languages raw text tokens transfer type zero-shot