STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Models

arXiv:2312.09040v2 Announce Type: replace-cross
Abstract: Albeit great performance of Transformer-based speech selfsupervised learning (SSL) models, their large parameter size and computational cost make them unfavorable to utilize. In this study, we propose to compress the speech SSL models by distilling speech temporal relation (STaR). Unlike previous works that directly match the representation for each speech frame, STaR distillation transfers temporal relation between speech frames, which is more suitable for lightweight student with limited capacity. We explore three STaR distillation objectives …

abstract arxiv computational cost cs.cl cs.sd eess.as performance self-supervised learning speech ssl star study supervised learning temporal them transformer type