Introducing Semantics into Speech Encoders. (arXiv:2211.08402v1 [cs.CL])

Recent studies find existing self-supervised speech encoders contain
primarily acoustic rather than semantic information. As a result, pipelined
supervised automatic speech recognition (ASR) to large language model (LLM)
systems achieve state-of-the-art results on semantic spoken language tasks by
utilizing rich semantic representations from the LLM. These systems come at the
cost of labeled audio transcriptions, which is expensive and time-consuming to
obtain. We propose a task-agnostic unsupervised way of incorporating semantic
information from LLMs into self-supervised speech encoders without labeled …

arxiv semantics speech