${M^2D}$NeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields

arXiv:2405.05010v1 Announce Type: new
Abstract: Neural fields (NeRF) have emerged as a promising approach for representing continuous 3D scenes. Nevertheless, the lack of semantic encoding in NeRFs poses a significant challenge for scene decomposition. To address this challenge, we present a single model, Multi-Modal Decomposition NeRF (${M^2D}$NeRF), that is capable of both text-based and visual patch-based edits. Specifically, we use multi-modal feature distillation to integrate teacher features from pretrained visual and language models into 3D semantic feature volumes, thereby facilitating …

3d scenes abstract arxiv challenge continuous cs.cv encoding feature fields modal multi-modal nerf semantic type