Robust Pronoun Use Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?

arXiv:2404.03134v1 Announce Type: new
Abstract: Robust, faithful and harm-free pronoun use for individuals is an important goal for language models as their use increases, but prior work tends to study only one or two of these components at a time. To measure progress towards the combined goal, we introduce the task of pronoun use fidelity: given a context introducing a co-referring entity and pronoun, the task is to reuse the correct pronoun later, independent of potential distractors. We present a …

abstract arxiv components cs.cl cs.cy english fidelity free harm language language models llms prior progress reasoning robust study type work