SalFoM: Dynamic Saliency Prediction with Video Foundation Models

arXiv:2404.03097v1 Announce Type: new
Abstract: Recent advancements in video saliency prediction (VSP) have shown promising performance compared to the human visual system, whose emulation is the primary goal of VSP. However, current state-of-the-art models employ spatio-temporal transformers trained on limited amounts of data, hindering generalizability adaptation to downstream tasks. The benefits of vision foundation models present a potential solution to improve the VSP process. However, adapting image foundation models to the video domain presents significant challenges in modeling scene dynamics …

abstract art arxiv benefits cs.cv current data dynamic foundation however human performance prediction state state-of-the-art models tasks temporal transformers type video visual