COMPS: Conceptual Minimal Pair Sentences for testing Property Knowledge and Inheritance in Pre-trained Language Models. (arXiv:2210.01963v2 [cs.CL] UPDATED)

A characteristic feature of human semantic memory is its ability to not only
store and retrieve the properties of concepts observed through experience, but
to also facilitate the inheritance of properties (can breathe) from
superordinate concepts (animal) to their subordinates (dog) -- i.e. demonstrate
property inheritance. In this paper, we present COMPS, a collection of minimal
pair sentences that jointly tests pre-trained language models (PLMs) on their
ability to attribute properties to concepts and their ability to demonstrate
property inheritance …

arxiv knowledge language language models testing