Exploring the Joint Use of Rehearsal and Knowledge Distillation in Continual Learning for Spoken Language Understanding. (arXiv:2211.08161v1 [eess.AS])

Continual learning refers to a dynamical framework in which a model or agent
receives a stream of non-stationary data over time and must adapt to new data
while preserving previously acquired knowledge. Unfortunately, deep neural
networks fail to meet these two desiderata, incurring the so-called
catastrophic forgetting phenomenon. Whereas a vast array of strategies have
been proposed to attenuate forgetting in the computer vision domain, for
speech-related tasks, on the other hand, there is a dearth of works. In this …

arxiv continual distillation knowledge language spoken language understanding understanding