GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration

arXiv:2311.12015v2 Announce Type: replace-cross
Abstract: We introduce a pipeline that enhances a general-purpose Vision Language Model, GPT-4V(ision), to facilitate one-shot visual teaching for robotic manipulation. This system analyzes videos of humans performing tasks and outputs executable robot programs that incorporate insights into affordances. The process begins with GPT-4V analyzing the videos to obtain textual explanations of environmental and action details. A GPT-4-based task planner then encodes these details into a symbolic task plan. Subsequently, vision systems spatially and temporally ground …

abstract arxiv cs.cl cs.cv cs.ro general gpt gpt-4v human humans insights language language model manipulation multimodal pipeline planning process robot robotic robotic manipulation robotics tasks teaching type videos vision vision language model visual