Quantifying the Plausibility of Context Reliance in Neural Machine Translation

arXiv:2310.01188v2 Announce Type: replace-cross
Abstract: Establishing whether language models can use contextual information in a human-plausible way is important to ensure their trustworthiness in real-world settings. However, the questions of when and which parts of the context affect model generations are typically tackled separately, with current plausibility evaluations being practically limited to a handful of artificial benchmarks. To address this, we introduce Plausibility Evaluation of Context Reliance (PECoRe), an end-to-end interpretability framework designed to quantify context usage in language models' …

abstract arxiv context cs.ai cs.cl cs.hc cs.lg current however human information language language models machine machine translation neural machine translation questions reliance translation type world