Learning by Aligning 2D Skeleton Sequences and Multi-Modality Fusion

arXiv:2305.19480v5 Announce Type: replace
Abstract: This paper presents a self-supervised temporal video alignment framework which is useful for several fine-grained human activity understanding applications. In contrast with the state-of-the-art method of CASA, where sequences of 3D skeleton coordinates are taken directly as input, our key idea is to use sequences of 2D skeleton heatmaps as input. Unlike CASA which performs self-attention in the temporal domain only, we feed 2D skeleton heatmaps to a video transformer which performs self-attention both in …

abstract alignment applications art arxiv contrast cs.cv fine-grained framework fusion human key paper state temporal type understanding video