Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning

arXiv:2312.13980v2 Announce Type: replace
Abstract: Multi-view diffusion models, obtained by applying Supervised Finetuning (SFT) to text-to-image diffusion models, have driven recent breakthroughs in text-to-3D research. However, due to the limited size and quality of existing 3D datasets, they still suffer from multi-view inconsistencies and Neural Radiance Field (NeRF) reconstruction artifacts. We argue that multi-view diffusion models can benefit from further Reinforcement Learning Finetuning (RLFT), which allows models to learn from the data generated by themselves and improve beyond their dataset …

arxiv cs.cv diffusion diffusion models finetuning improving type view