A Multimodal Approach to Device-Directed Speech Detection with Large Language Models

arXiv:2403.14438v1 Announce Type: cross
Abstract: Interactions with virtual assistants typically start with a predefined trigger phrase followed by the user command. To make interactions with the assistant more intuitive, we explore whether it is feasible to drop the requirement that users must begin each command with a trigger phrase. We explore this task in three ways: First, we train classifiers using only acoustic information obtained from the audio waveform. Second, we take the decoder outputs of an automatic speech recognition …

abstract arxiv assistant assistants command cs.cl cs.lg detection eess.as explore interactions language language models large language large language models multimodal speech type virtual virtual assistants