[D] Why use hard-coded tokenization in NLP instead of a learned tokenization?

Usually when you are going to create an NLP model, you use some library that performs tokenization of the text. The network receives these tokens on input, and on output, it has to predict which token (class) is the most likely.
Why not use a prior network that on input receives the raw text, and generate an learned output to the main network. I believe that letting the neural network itself tokenize the text is the best way to process …

machinelearning nlp tokenization