The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

arXiv:2309.12288v3 Announce Type: replace-cross
Abstract: We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form "A is B", it will not automatically generalize to the reverse direction "B is A". This is the Reversal Curse. For instance, if a model is trained on "Valentina Tereshkova was the first woman to travel to space", it will not automatically be able to answer the question, "Who was the …

abstract arxiv auto cs.ai cs.cl cs.lg failure form language language models large language large language models learn llms type will