Predicting Long-horizon Futures by Conditioning on Geometry and Time

arXiv:2404.11554v1 Announce Type: new
Abstract: Our work explores the task of generating future sensor observations conditioned on the past. We are motivated by `predictive coding' concepts from neuroscience as well as robotic applications such as self-driving vehicles. Predictive video modeling is challenging because the future may be multi-modal and learning at scale remains computationally expensive for video processing. To address both challenges, our key insight is to leverage the large-scale pretraining of image diffusion models which can handle multi-modality. We …

abstract applications arxiv coding concepts cs.cv driving future futures geometry horizon modal modeling multi-modal neuroscience predictive robotic self-driving self-driving vehicles sensor type vehicles video work