We're Calling an Intervention: Taking a Closer Look at Language Model Adaptation to Different Types of Linguistic Variation

arXiv:2404.07304v1 Announce Type: new
Abstract: We present a suite of interventions and experiments that allow us to understand language model adaptation to text with linguistic variation (e.g., nonstandard or dialectal text). Our interventions address several features of linguistic variation, resulting in character, subword, and word-level changes. Applying our interventions during language model adaptation with varying size and nature of training data, we gain important insights into what makes linguistic variation particularly difficult for language models to deal with. For instance, …

abstract arxiv closer look cs.cl features language language model look model adaptation text type types variation