Discovering Novel Actions from Open World Egocentric Videos with Object-Grounded Visual Commonsense Reasoning

arXiv:2305.16602v2 Announce Type: replace
Abstract: Learning to infer labels in an open world, i.e., in an environment where the target ``labels'' are unknown, is an important characteristic for achieving autonomy. Foundation models, pre-trained on enormous amounts of data, have shown remarkable generalization skills through prompting, particularly in zero-shot inference. However, their performance is restricted to the correctness of the target label's search space, i.e., candidate labels provided in the prompt. This target search space can be unknown or exceptionally large …

abstract arxiv autonomy commonsense cs.cv data environment foundation labels novel object prompting reasoning skills through type videos visual world