FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing

arXiv:2310.05922v2 Announce Type: replace
Abstract: Text-to-video editing aims to edit the visual appearance of a source video conditional on textual prompts. A major challenge in this task is to ensure that all frames in the edited video are visually consistent. Most recent works apply advanced text-to-image diffusion models to this task by inflating 2D spatial attention in the U-Net into spatio-temporal attention. Although temporal context can be added through spatio-temporal attention, it may introduce some irrelevant information for each patch …

abstract advanced apply arxiv attention challenge consistent cs.cv diffusion diffusion models edit editing flatten flow image image diffusion major optical optical flow prompts text text-to-image text-to-video textual type video visual