Retrieval is Accurate Generation

arXiv:2402.17532v1 Announce Type: new
Abstract: Standard language models generate text by selecting tokens from a fixed, finite, and standalone vocabulary. We introduce a novel method that selects context-aware phrases from a collection of supporting documents. One of the most significant challenges for this paradigm shift is determining the training oracles, because a string of text can be segmented in various ways and each segment can be retrieved from numerous possible documents. To address this, we propose to initialize the training …

abstract arxiv challenges collection context cs.cl documents generate language language models novel paradigm retrieval shift standard string text tokens training type