Understanding Catastrophic Forgetting in Language Models via Implicit Inference

arXiv:2309.10105v2 Announce Type: replace-cross
Abstract: We lack a systematic understanding of the effects of fine-tuning (via methods such as instruction-tuning or reinforcement learning from human feedback), particularly on tasks outside the narrow fine-tuning distribution. In a simplified scenario, we demonstrate that improving performance on tasks within the fine-tuning data distribution comes at the expense of capabilities on other tasks. We hypothesize that language models implicitly infer the task of the prompt and that fine-tuning skews this inference towards tasks in …

abstract arxiv catastrophic forgetting cs.cl cs.lg data distribution effects feedback fine-tuning human human feedback improving inference language language models narrow performance reinforcement reinforcement learning simplified tasks type understanding via