A low latency attention module for streaming self-supervised speech representation learning

arXiv:2302.13451v2 Announce Type: replace-cross
Abstract: The transformer is a fundamental building block in deep learning, and the attention mechanism is the transformer's core component. Self-supervised speech representation learning (SSRL) represents a popular use-case for the transformer architecture. Due to transformers' acausal behavior, the use of transformers for SSRL has been predominantly focused on acausal applications. However, several media processing problems, such as speech processing, require real-time solutions. In this paper, we present an implementation of the attention module that enables …

abstract architecture arxiv attention behavior block building case core cs.cl cs.lg cs.sd deep learning eess.as latency low low latency popular representation representation learning speech streaming transformer transformer architecture transformers type