RAFT: Adapting Language Model to Domain Specific RAG

arXiv:2403.10131v1 Announce Type: new
Abstract: Pretraining Large Language Models (LLMs) on large corpora of textual data is now a standard paradigm. When using these LLMs for many downstream applications, it is common to additionally bake in new knowledge (e.g., time-critical news, or private domain knowledge) into the pretrained model either through RAG-based-prompting, or fine-tuning. However, the optimal methodology for the model to gain such new knowledge remains an open question. In this paper, we present Retrieval Augmented FineTuning (RAFT), a …

abstract applications arxiv cs.ai cs.cl data domain domain knowledge knowledge language language model language models large language large language models llms paradigm pretraining prompting raft rag standard textual through type