LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning

arXiv:2406.11815v1 Announce Type: cross
Abstract: In recent years, instruction-tuned Large Multimodal Models (LMMs) have been successful at several tasks, including image captioning and visual question answering; yet leveraging these models remains an open question for robotics. Prior LMMs for robotics applications have been extensively trained on language and action data, but their ability to generalize in different settings has often been less than desired. To address this, we introduce LLARVA, a model trained with a novel instruction tuning method that …

abstract action applications arxiv captioning cs.cv cs.lg cs.ro data image instruction-tuned instruction tuning language large multimodal models lmms multimodal multimodal models prior question question answering robot robotics tasks tuning type vision visual