HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision

arXiv:2404.09933v1 Announce Type: new
Abstract: Large Vision Language Models (VLMs) are now the de facto state-of-the-art for a number of tasks including visual question answering, recognising objects, and spatial referral. In this work, we propose the HOI-Ref task for egocentric images that aims to understand interactions between hands and objects using VLMs. To enable HOI-Ref, we curate the HOI-QA dataset that consists of 3.9M question-answer pairs for training and evaluating VLMs. HOI-QA includes questions relating to locating hands, objects, and …

abstract art arxiv cs.cv images interactions language language models object objects question question answering spatial state tasks type vision visual vlms work