Hands-Free VR

arXiv:2402.15083v1 Announce Type: cross
Abstract: The paper introduces Hands-Free VR, a voice-based natural-language interface for VR. The user gives a command using their voice, the speech audio data is converted to text using a speech-to-text deep learning model that is fine-tuned for robustness to word phonetic similarity and to spoken English accents, and the text is mapped to an executable VR command using a large language model that is robust to natural language diversity. Hands-Free VR was evaluated in a …

abstract accents arxiv audio command cs.ai cs.cl cs.hc data deep learning english free language mapped natural paper robustness speech speech-to-text spoken text type voice word