AmbigDocs: Reasoning across Documents on Different Entities under the Same Name

arXiv:2404.12447v1 Announce Type: new
Abstract: Different entities with the same name can be difficult to distinguish. Handling confusing entity mentions is a crucial skill for language models (LMs). For example, given the question "Where was Michael Jordan educated?" and a set of documents discussing different people named Michael Jordan, can LMs distinguish entity mentions to generate a cohesive answer to the question? To test this ability, we introduce a new benchmark, AmbigDocs. By leveraging Wikipedia's disambiguation pages, we identify a …

abstract arxiv cs.cl documents example language language models lms michael jordan people question reasoning set type