MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception

arXiv:2312.07472v3 Announce Type: replace
Abstract: It is a long-lasting goal to design an embodied system that can solve long-horizon open-world tasks in human-like ways. However, existing approaches usually struggle with compound difficulties caused by the logic-aware decomposition and context-aware execution of these tasks. To this end, we introduce MP5, an open-ended multimodal embodied system built upon the challenging Minecraft simulator, which can decompose feasible sub-objectives, design sophisticated situation-aware plans, and perform embodied action control, with frequent communication with a goal-conditioned …

abstract arxiv context cs.cv design embodied horizon human human-like logic minecraft modal multi-modal open-world perception solve struggle tasks type via world