RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion

arXiv:2404.07199v1 Announce Type: cross
Abstract: We introduce RealmDreamer, a technique for generation of general forward-facing 3D scenes from text descriptions. Our technique optimizes a 3D Gaussian Splatting representation to match complex text prompts. We initialize these splats by utilizing the state-of-the-art text-to-image generators, lifting their samples into 3D, and computing the occlusion volume. We then optimize this representation across multiple views as a 3D inpainting task with image-conditional diffusion models. To learn correct geometric structure, we incorporate a depth diffusion …

3d scene generation 3d scenes abstract art arxiv computing cs.ai cs.cv cs.gr cs.lg diffusion general generators image image generators inpainting match prompts representation samples state text text-to-image type