PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation

arXiv:2404.13026v1 Announce Type: new
Abstract: Realistic object interactions are crucial for creating immersive virtual experiences, yet synthesizing realistic 3D object dynamics in response to novel interactions remains a significant challenge. Unlike unconditional or text-conditioned dynamics generation, action-conditioned dynamics requires perceiving the physical material properties of objects and grounding the 3D motion prediction on these properties, such as object stiffness. However, estimating physical material properties is an open problem due to the lack of material ground-truth data, as measuring these properties …

3d objects arxiv cs.ai cs.cv objects physics type via video video generation