Lumiere: A Space-Time Diffusion Model for Video Generation (Paper Explained)

#lumiere #texttovideoai #google

LUMIERE by Google Research tackles globally consistent text-to-video generation by extending the U-Net downsampling concept to the temporal axis of videos.

OUTLINE:
0:00 - Introduction
8:20 - Problems with keyframes
16:55 - Space-Time U-Net (STUNet)
21:20 - Extending U-Nets to video
37:20 - Multidiffusion for SSR prediction fusing
44:00 - Stylized generation by swapping weights
49:15 - Training & Evaluation
53:20 - Societal Impact & Conclusion


Paper: https://arxiv.org/abs/2401.12945
Website: https://lumiere-video.github.io/

Abstract:
We introduce Lumiere -- a text-to-video …

concept consistent diffusion diffusion model downsampling explained google google lumiere google research introduction lumiere paper prediction research space ssr temporal text text-to-video video video generation videos