Selective Visual Representations Improve Convergence and Generalization for Embodied AI

arXiv:2311.04193v2 Announce Type: replace
Abstract: Embodied AI models often employ off the shelf vision backbones like CLIP to encode their visual observations. Although such general purpose representations encode rich syntactic and semantic information about the scene, much of this information is often irrelevant to the specific task at hand. This introduces noise within the learning process and distracts the agent's focus from task-relevant visual cues. Inspired by selective attention in humans-the process through which people filter their perception based on …

arxiv convergence cs.ai cs.cv embodied embodied ai type visual