Scaling multimodal understanding to long videos

When building machine learning models for real-life applications, we need to consider inputs from multiple modalities in order to capture various aspects of the world around us. For example, audio, video, and text all provide varied and complementary information about a visual input. However, building multimodal models is challenging due to the heterogeneity of the modalities. Some of the modalities might be well synchronized in …

ai applications audio building computer vision deepmind engineer example google google deepmind google research information life machine machine learning machine learning models multimodal multimodal learning multiple research research scientist scaling software software engineer text understanding video video analysis videos world