A Family of Pretrained Transformer Language Models for Russian

arXiv:2309.10931v2 Announce Type: replace
Abstract: Transformer language models (LMs) are fundamental to NLP research methodologies and applications in various languages. However, developing such models specifically for the Russian language has received little attention. This paper introduces a collection of 13 Russian Transformer LMs, which spans encoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder (ruT5, FRED-T5) architectures. We provide a report on the model architecture design and pretraining, and the results of evaluating their generalization abilities on Russian language understanding and …

abstract applications arxiv attention collection cs.cl decoder encoder family however language language models languages lms nlp paper research transformer transformer language models type