RA-DIT: Retrieval-Augmented Dual Instruction Tuning

arXiv:2310.01352v4 Announce Type: replace
Abstract: Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tuning steps: …

abstract arxiv build cs.ai cs.cl data data store data stores external data instruction tuning integration knowledge language language models leads performance pre-training retrieval retrieval-augmented store stores training type