Bottleneck Low-rank Transformers for Low-resource Spoken Language Understanding. (arXiv:2206.14318v1 [cs.CL])

End-to-end spoken language understanding (SLU) systems benefit from
pretraining on large corpora, followed by fine-tuning on application-specific
data. The resulting models are too large for on-edge applications. For
instance, BERT-based systems contain over 110M parameters. Observing the model
is overparameterized, we propose lean transformer structure where the dimension
of the attention mechanism is automatically reduced using group sparsity. We
propose a variant where the learned attention subspace is transferred to an
attention bottleneck layer. In a low-resource setting and without …

arxiv language spoken language understanding transformers understanding