Acquiring Linguistic Knowledge from Multimodal Input

arXiv:2402.17936v1 Announce Type: new
Abstract: In contrast to children, language models (LMs) exhibit considerably inferior data efficiency when acquiring language. In this submission to the BabyLM Challenge (Warstadt et al., 2023), we test the hypothesis that this data efficiency gap is partly caused by a lack of multimodal input and grounding in the learning environment of typical language models. Although previous work looking into this question found that multimodal training can even harm language-only performance, we speculate that these findings …

abstract arxiv challenge children contrast cs.cl data efficiency gap hypothesis knowledge language language models lms multimodal test type