Tokenizer Choice For LLM Training: Negligible or Crucial?

arXiv:2310.08754v4 Announce Type: replace
Abstract: The recent success of Large Language Models (LLMs) has been predominantly driven by curating the training dataset composition, scaling of model architectures and dataset sizes and advancements in pretraining objectives, leaving tokenizer influence as a blind spot. Shedding light on this underexplored area, we conduct a comprehensive study on the influence of tokenizer choice on LLM downstream performance by training 24 mono- and multilingual LLMs at a 2.6B parameter scale, ablating different tokenizer algorithms and …

abstract architectures arxiv blind blind spot cs.lg dataset influence language language models large language large language models light llm llms pretraining scaling spot success training type