BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models

arXiv:2404.04113v1 Announce Type: new
Abstract: Knowledge probing assesses to which degree a language model (LM) has successfully learned relational knowledge during pre-training. Probing is an inexpensive way to compare LMs of different sizes and training configurations. However, previous approaches rely on the objective function used in pre-training LMs and are thus applicable only to masked or causal LMs. As a result, comparing different types of LMs becomes impossible. To address this, we propose an approach that uses an LM's inherent …

abstract arxiv causal cs.cl framework function however knowledge language language model language models lms pre-training relational training type