Grounding Spatial Relations in Text-Only Language Models

arXiv:2403.13666v1 Announce Type: new
Abstract: This paper shows that text-only Language Models (LM) can learn to ground spatial relations like "left of" or "below" if they are provided with explicit location information of objects and they are properly trained to leverage those locations. We perform experiments on a verbalized version of the Visual Spatial Reasoning (VSR) dataset, where images are coupled with textual statements which contain real or fake spatial relations between two objects of the image. We verbalize the …

abstract arxiv cs.cl information language language models learn location locations objects paper relations shows spatial text type visual