Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL

arXiv:2404.09857v1 Announce Type: new
Abstract: Embodied visual tracking is to follow a target object in dynamic 3D environments using an agent's egocentric vision. This is a vital and challenging skill for embodied agents. However, existing methods suffer from inefficient training and poor generalization. In this paper, we propose a novel framework that combines visual foundation models (VFM) and offline reinforcement learning (offline RL) to empower embodied visual tracking. We use a pre-trained VFM, such as ``Tracking Anything", to extract semantic …

abstract agent agents arxiv cs.ai cs.cv cs.ro dynamic embodied environments foundation however object offline paper tracking training type vision visual vital