Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering

arXiv:2307.16877v2 Announce Type: replace
Abstract: Retriever-augmented instruction-following models are attractive alternatives to fine-tuned approaches for information-seeking tasks such as question answering (QA). By simply prepending retrieved documents in its input along with an instruction, these models can be adapted to various information domains and tasks without additional fine-tuning. While the model responses tend to be natural and fluent, the additional verbosity makes traditional QA evaluation metrics such as exact match (EM) and F1 unreliable for accurately quantifying model performance.
In …

abstract arxiv cs.ai cs.cl documents domains fine-tuning information question question answering retriever tasks type