Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Language. (arXiv:2201.10262v1 [cs.CL])

Web: http://arxiv.org/abs/2201.10262

With the methodological support of probing (or diagnostic classification),
recent studies have demonstrated that Transformers encode syntactic and
semantic information to some extent. Following this line of research, this
paper aims at taking semantic probing to an abstraction extreme with the goal
of answering the following research question: can contemporary
Transformer-based models reflect an underlying Foundational Ontology? To this
end, we present a systematic Foundational Ontology (FO) probing methodology to
investigate whether Transformers-based models encode abstract semantic
information. Following different …

arxiv language natural natural language transformers