JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition

arXiv:2403.18843v1 Announce Type: cross
Abstract: Visual Speech Recognition (VSR) tasks are generally recognized to have a lower theoretical performance ceiling than Automatic Speech Recognition (ASR), owing to the inherent limitations of conveying semantic information visually. To mitigate this challenge, this paper introduces an advanced knowledge distillation approach using a Joint-Embedding Predictive Architecture (JEPA), named JEP-KD, designed to more effectively utilize audio features during model training. Central to JEP-KD is the inclusion of a generative network within the embedding layer, which …

abstract advanced architecture arxiv asr automatic speech recognition challenge cs.cl cs.cv cs.lg cs.sd distillation eess.as embedding information knowledge limitations paper performance predictive recognition semantic speech speech recognition tasks type visual