I pretrained 16 language models from scratch with different tokenizers to benchmark the difference. Here are the results. [Research]

I'm the author of [TokenMonster](https://github.com/alasdairforsythe/tokenmonster), a free open-source tokenizer and vocabulary builder. I've posted on here a few times as the project has evolved, and each time I'm asked "have you tested it on a language model?".

Well here it is. I spent $8,000 from my own pocket, and 2 months, pretraining from scratch, finetuning and evaluating 16 language models. 12 small sized models of 91 - 124M parameters, and 4 medium sized models of 354M parameters.

[Here is the …

finetuning gpt gpt-2 language language models machinelearning medium small summary