AI2 Introduces Efficient Hierarchical Domain Adaptation for Pretrained Language Models

Many natural language processing (NLP) benchmarks have seen significant advancement thanks to pretrained language models (PLMs) trained on large general-domain corpora. 

Sparse models that combine expertise have recently been presented as a way to improve training efficiency. Individual domains are often assumed to be unique in previous work, and models are created accordingly. Because the parameters grow linearly with the domains, this approach does not scale well across several domains. It also prevents sharing representations between related domains during training …

ai2 domain adaptation hierarchical language language models languagetechnology