DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos

arXiv:2405.02280v1 Announce Type: new
Abstract: Existing VLMs can track in-the-wild 2D video objects while current generative models provide powerful visual priors for synthesizing novel views for the highly under-constrained 2D-to-3D object lifting. Building upon this exciting progress, we present DreamScene4D, the first approach that can generate three-dimensional dynamic scenes of multiple objects from monocular in-the-wild videos with large object motion across occlusions and novel viewpoints. Our key insight is to design a "decompose-then-recompose" scheme to factorize both the whole video …

2d-to-3d 3d object abstract arxiv building cs.cv current dynamic generate generative generative models multiple novel object objects progress three-dimensional type video videos visual vlms while