Combating Missing Modalities in Egocentric Videos at Test Time

arXiv:2404.15161v1 Announce Type: new
Abstract: Understanding videos that contain multiple modalities is crucial, especially in egocentric videos, where combining various sensory inputs significantly improves tasks like action recognition and moment localization. However, real-world applications often face challenges with incomplete modalities due to privacy concerns, efficiency needs, or hardware issues. Current methods, while effective, often necessitate retraining the model entirely to handle missing modalities, making them computationally intensive, particularly with large training datasets. In this study, we propose a novel approach …

abstract action recognition applications arxiv challenges concerns cs.cv current efficiency face hardware however inputs localization moment multiple privacy recognition sensory tasks test type understanding videos world