On the Utility of 3D Hand Poses for Action Recognition

arXiv:2403.09805v1 Announce Type: cross
Abstract: 3D hand poses are an under-explored modality for action recognition. Poses are compact yet informative and can greatly benefit applications with limited compute budgets. However, poses alone offer an incomplete understanding of actions, as they cannot fully capture objects and environments with which humans interact. To efficiently model hand-object interactions, we propose HandFormer, a novel multimodal transformer. HandFormer combines 3D hand poses at a high temporal resolution for fine-grained motion modeling with sparsely sampled RGB …

abstract action recognition applications arxiv benefit budgets compute cs.cv cs.lg environments however humans objects recognition type understanding utility