Towards Understanding the Relationship between In-context Learning and Compositional Generalization

arXiv:2403.11834v1 Announce Type: cross
Abstract: According to the principle of compositional generalization, the meaning of a complex expression can be understood as a function of the meaning of its parts and of how they are combined. This principle is crucial for human language processing and also, arguably, for NLP models in the face of out-of-distribution data. However, many neural network models, including Transformers, have been shown to struggle with compositional generalization. In this paper, we hypothesize that forcing models to …

abstract arxiv context cs.cl cs.lg function human in-context learning language language processing meaning nlp nlp models processing relationship type understanding