Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining

arXiv:2404.05428v1 Announce Type: new
Abstract: The world of language models is going through turbulent times, better and ever larger models are coming out at an unprecedented speed. However, we argue that, especially for the scientific community, encoder models of up to 1 billion parameters are still very much needed, their primary usage being in enriching large collections of data with metadata necessary for downstream research. We investigate the best way to ensure the existence of such encoder models on the …

abstract arxiv billion community cost cs.cl development diet encoder however language language models languages larger models parameters pretraining scientific speed through type via world