Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation

arXiv:2404.01518v1 Announce Type: cross
Abstract: We propose a novel approach to the action segmentation task for long, untrimmed videos, based on solving an optimal transport problem. By encoding a temporal consistency prior into a Gromov-Wasserstein problem, we are able to decode a temporally consistent segmentation from a noisy affinity/matching cost matrix between video frames and action classes. Unlike previous approaches, our method does not require knowing the action order for a video to attain temporal consistency. Furthermore, our resulting (fused) …

abstract arxiv consistent cost cs.cv cs.lg decode eess.iv encoding matrix novel prior segmentation temporal transport type unsupervised videos