[D] Preprocessing of Wikipedia Dumps for Language Modeling from Scratch

I want to train a language model from scratch on wikipedia dumps of a language, say French. I download the dumps and extract them using the wikiextractor tool. I lower-case everything but keep all the accents, since they are important for French. So far so good, but now it gets blurry.

There is very little information about the specifics of preprocessing people are applying to the dumps before training tokenizers and feeding the data into the model.

1. How are section …

language machinelearning modeling wikipedia