A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction

arXiv:2310.20225v2 Announce Type: replace
Abstract: People with blindness and low vision (pBLV) encounter substantial challenges when it comes to comprehensive scene recognition and precise object identification in unfamiliar environments. Additionally, due to the vision loss, pBLV have difficulty in accessing and identifying potential tripping hazards on their own. In this paper, we present a pioneering approach that leverages a large vision-language model to enhance visual perception for pBLV, offering detailed and comprehensive descriptions of the surrounding environments and providing warnings …

abstract arxiv blindness challenges cs.ai cs.cv environmental environments foundation foundation model hazards identification loss low modal multi-modal object people recognition type vision