[D] What happened to huggingface tokenizers API?

Tokenizers library used to be very nice to use. It had one main class `Tokenizer` with all of the parameters in its constructor and with all necessary methods like `.train()`, `.encode()` and `.decode()` at hand. It provided reasonable defaults and allowed for customization if needed.

Now it is a complete mess. To train a tokenizer I now have to create instances of, like, 5-6 classes: `PreTokenizer`, `Model`, `Tokenizer`, `Trainer`, `Decoder`... It is quite difficult to understand what variants of those …

api customization decode encode huggingface library machinelearning nice