Can You Learn Semantics Through Next-Word Prediction? The Case of Entailment

arXiv:2402.13956v1 Announce Type: new
Abstract: Do LMs infer the semantics of text from co-occurrence patterns in their training data? Merrill et al. (2022) argue that, in theory, probabilities predicted by an optimal LM encode semantic information about entailment relations, but it is unclear whether neural LMs trained on corpora learn entailment in this way because of strong idealizing assumptions made by Merrill et al. In this work, we investigate whether their theory can be used to decode entailment judgments from …

abstract arxiv case cs.cl data encode information learn lms next patterns prediction relations semantic semantics text theory through training training data type word