Zero-shot Prompt-based Video Encoder for Surgical Gesture Recognition

arXiv:2403.19786v1 Announce Type: new
Abstract: Purpose: Surgical video is an important data stream for gesture recognition. Thus, robust visual encoders for those data-streams is similarly important. Methods: Leveraging the Bridge-Prompt framework, we fine-tune a pre-trained vision-text model (CLIP) for gesture recognition in surgical videos. This can utilize extensive outside video data such as text, but also make use of label meta-data and weakly supervised contrastive losses. Results: Our experiments show that prompt-based video encoder outperforms standard encoders in surgical gesture …

abstract arxiv bridge clip cs.cv data data stream encoder framework gesture recognition prompt recognition robust text type video video data videos vision visual zero-shot