More Room for Language: Investigating the Effect of Retrieval on Language Models

arXiv:2404.10939v1 Announce Type: new
Abstract: Retrieval-augmented language models pose a promising alternative to standard language modeling. During pretraining, these models search in a corpus of documents for contextually relevant information that could aid the language modeling objective. We introduce an 'ideal retrieval' methodology to study these models in a fully controllable setting. We conduct an extensive evaluation to examine how retrieval augmentation affects the behavior of the underlying language model. Among other things, we observe that these models: i) save …

abstract arxiv cs.cl documents information language language models methodology modeling pretraining retrieval retrieval-augmented room search standard study type