[D] Creating your own language model (i.e., text encoder) for a specific domain. When is it worth it and what should you be aware of?

I recently made custom BERT and ELECTRA models for the fashion domain that could also handle English and my own native language (I'm not in the US). I noticed that performance wasn't as good as I anticipated and felt that it wasn't worth it.

Are there any papers or resources regarding when it's worth it to create your own pre-trained LM from scratch? I recall reading a paper for the biomedical domain a long time ago titled [_Pretrained Language Models …

bert domain encoder english fashion good language language model machinelearning performance text