ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots

arXiv:2209.08199v2 Announce Type: replace-cross
Abstract: We present a new task and dataset, ScreenQA, for screen content understanding via question answering. The existing screen datasets are focused either on structure and component-level understanding, or on a much higher-level composite task such as navigation and task completion. We attempt to bridge the gap between these two by annotating 86K question-answer pairs over the RICO dataset in hope to benchmark the screen reading comprehension capacity.

abstract app arxiv bridge cs.cl cs.cv cs.hc dataset datasets gap mobile mobile app navigation question question answering scale type understanding via