Learning Object States from Actions via Large Language Models

arXiv:2405.01090v1 Announce Type: new
Abstract: Temporally localizing the presence of object states in videos is crucial in understanding human activities beyond actions and objects. This task has suffered from a lack of training data due to object states' inherent ambiguity and variety. To avoid exhaustive annotation, learning from transcribed narrations in instructional videos would be intriguing. However, object states are less described in narrations compared to actions, making them less effective. In this work, we propose to extract the object …

abstract annotation arxiv beyond cs.cv data human language language models large language large language models object objects training training data type understanding via videos