Spectral Motion Alignment for Video Motion Transfer using Diffusion Models

arXiv:2403.15249v1 Announce Type: cross
Abstract: The evolution of diffusion models has greatly impacted video generation and understanding. Particularly, text-to-video diffusion models (VDMs) have significantly facilitated the customization of input video with target appearance, motion, etc. Despite these advances, challenges persist in accurately distilling motion information from video frames. While existing works leverage the consecutive frame residual as the target motion vector, they inherently lack global motion context and are vulnerable to frame-wise distortions. To address this, we present Spectral Motion …

abstract advances alignment arxiv challenges cs.ai cs.cv cs.lg customization diffusion diffusion models etc evolution information text text-to-video transfer type understanding video video diffusion video generation