RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis

arXiv:2402.16117v1 Announce Type: cross
Abstract: Robotic behavior synthesis, the problem of understanding multimodal inputs and generating precise physical control for robots, is an important part of Embodied AI. Despite successes in applying multimodal large language models for high-level understanding, it remains challenging to translate these conceptual understandings into detailed robotic actions while achieving generalization across various scenarios. In this paper, we propose a tree-structured multimodal code generation framework for generalized robotic behavior synthesis, termed RoboCodeX. RoboCodeX decomposes high-level human instructions …

abstract arxiv behavior code code generation control cs.ai cs.cv cs.ro embodied embodied ai inputs language language models large language large language models multimodal part robotic robots synthesis translate type understanding