[P] Testing different popular GPT tokenizers

I made a small project for testing if different popular tokenizers are lossless. I.e. do they give back the original input after encode+decode.

Turns out most of them are not.

https://github.com/skeskinen/hf-tokenizer-testing

Does it matter if tokenizers can/can't reproduce the input exactly? I guess this is subjective, but I'd say it's at least a nice feature. A feature that (perhaps surprisingly?) most tokenizers out there don't seem to have.

I wrote this for myself on a quest to find a tokenizer …

decode encode feature gpt least machinelearning nice popular project small testing