Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristics

arXiv:2402.12036v1 Announce Type: new
Abstract: Recent advances in pre-trained language modeling have facilitated significant progress across various natural language processing (NLP) tasks. Word masking during model training constitutes a pivotal component of language modeling in architectures like BERT. However, the prevalent method of word masking relies on random selection, potentially disregarding domain-specific linguistic attributes. In this article, we introduce an innovative masking approach leveraging genre and topicality information to tailor language models to specialized domains. Our method incorporates a ranking …

abstract advances architectures arxiv bert cs.cl domains language language model language processing masking model adaptation modeling natural natural language natural language processing nlp pivotal processing progress tasks through training type word