Understanding Text2Video through brand new Google's Lumiere paper

* **Full Article:** [**https://vishal-ai.medium.com/lumiere-a-space-time-diffusion-model-for-video-generation-dfff0b41b24a**](https://vishal-ai.medium.com/lumiere-a-space-time-diffusion-model-for-video-generation-dfff0b41b24a)

**1. Input and Initial Processing:**
— The input is a grayscale video with 3 frames.
— The video is downsampled both spatially and temporally to reduce resolution and number of frames, respectively.

**2. STUNet Layers and T2I Embedding Integration:**
***Convolution-based Inflation Blocks:***
— Applied at each downsampling layer, which includes 2D convolutions on spatial dimensions and 1D convolutions across the temporal dimension.
— After convolutions, T2I embeddings are integrated. These embeddings, derived from the textual description, …

convolution dimensions downsampling embedding inflation integration layer machinelearningnews processing reduce spatial video