Sequence to sequence pretraining for a less-resourced Slovenian language. (arXiv:2207.13988v1 [cs.CL])

Large pretrained language models have recently conquered the area of natural
language processing. As an alternative to predominant masked language modelling
introduced in BERT, the T5 model has introduced a more general training
objective, namely sequence to sequence transformation, which includes masked
language model but more naturally fits text generation tasks such as machine
translation, summarization, open-domain question answering, text
simplification, dialogue systems, etc. The monolingual variants of T5 models
have been limited to well-resourced languages, while the massively multilingual …

arxiv language sequence to sequence