Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs

arXiv:2402.14903v1 Announce Type: cross
Abstract: Tokenization, the division of input text into input tokens, is an often overlooked aspect of the large language model (LLM) pipeline and could be the source of useful or harmful inductive biases. Historically, LLMs have relied on byte pair encoding, without care to specific input domains. With the increased use of LLMs for reasoning, various number-specific tokenization schemes have been adopted, with popular models like LLaMa and PaLM opting for single-digit tokenization while GPT-3.5 and …

abstract arxiv biases cs.cl cs.lg encoding impact inductive language language model large language large language model llm llms pipeline text tokenization tokens type