Beyond MOT: Semantic Multi-Object Tracking

arXiv:2403.05021v1 Announce Type: new
Abstract: Current multi-object tracking (MOT) aims to predict trajectories of targets (i.e.,"where") in videos. Yet, knowing merely "where" is insufficient in many crucial applications. In comparison, semantic understanding such as fine-grained behaviors, interactions, and overall summarized captions (i.e., "what") from videos, associated with "where", is highly-desired for comprehensive video analysis. Thus motivated, we introduce Semantic Multi-Object Tracking (SMOT), that aims to estimate object trajectories and meanwhile understand semantic details of associated trajectories including instance captions, instance …

abstract analysis applications arxiv beyond captions comparison cs.cv current fine-grained interactions object semantic targets tracking type understanding video video analysis videos