Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling

arXiv:2403.14551v1 Announce Type: cross
Abstract: Today's most accurate language models are trained on orders of magnitude more language data than human language learners receive - but with no supervision from other sensory modalities that play a crucial role in human learning. Can we make LMs' representations and predictions more accurate (and more human-like) with more ecologically plausible supervision? This paper describes LexiContrastive Grounding (LCG), a grounded language learning procedure that leverages visual supervision to improve textual representations. LexiContrastive Grounding combines …

abstract arxiv cs.ai cs.cl cs.lg data human human-like language language data language models lms modeling orders predictions role sensory supervision type visual