Physically Grounded Vision-Language Models for Robotic Manipulation

arXiv:2309.02561v3 Announce Type: replace-cross
Abstract: Recent advances in vision-language models (VLMs) have led to improved performance on tasks such as visual question answering and image captioning. Consequently, these models are now well-positioned to reason about the physical world, particularly within domains such as robotic manipulation. However, current VLMs are limited in their understanding of the physical concepts (e.g., material, fragility) of common objects, which restricts their usefulness for robotic manipulation tasks that involve interaction and physical reasoning about such objects. …

arxiv cs.ai cs.cv cs.ro language language models manipulation robotic robotic manipulation type vision vision-language models