Greed is All You Need: An Evaluation of Tokenizer Inference Methods

arXiv:2403.01289v1 Announce Type: new
Abstract: While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabularies is often left unspecified, or ill-suited to the method in which they were constructed. We provide a controlled analysis of seven tokenizer inference methods across four different algorithms and three vocabulary sizes, performed on a novel intrinsic evaluation suite we curated for English, combining measures …

abstract arxiv build cs.cl decoding evaluation inference nlp nlp models text tokens type