Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis. (arXiv:2204.02448v1 [cs.HC])

We use a deep learning based approach to predict whether a selected element
in a mobile UI screenshot will be perceived by users as tappable, based on
pixels only instead of view hierarchies required by previous work. To help
designers better understand model predictions and to provide more actionable
design feedback than predictions alone, we additionally use ML interpretability
techniques to help explain the output of our model. We use XRAI to highlight
areas in the input screenshot that most …

analysis arxiv mobile modeling ui vision