Unifying Language Learning Paradigms. (arXiv:2205.05131v1 [cs.CL])

Web: http://arxiv.org/abs/2205.05131

Existing pre-trained models are generally geared towards a particular class
of problems. To date, there seems to be still no consensus on what the right
architecture and pre-training setup should be. This paper presents a unified
framework for pre-training models that are universally effective across
datasets and setups. We begin by disentangling architectural archetypes with
pre-training objectives -- two concepts that are commonly conflated. Next, we
present a generalized and unified perspective for self-supervision in NLP and
show how different …

arxiv language learning