Speech Understanding on Tiny Devices with A Learning Cache

arXiv:2311.18188v4 Announce Type: replace-cross
Abstract: This paper addresses spoken language understanding (SLU) on microcontroller-like embedded devices, integrating on-device execution with cloud offloading in a novel fashion. We leverage temporal locality in the speech inputs to a device and reuse recent SLU inferences accordingly. Our idea is simple: let the device match incoming inputs against cached results, and only offload inputs not matched to any cached ones to the cloud for full inference. Realization of this idea, however, is non-trivial: the …

abstract arxiv cache cloud cs.lg devices eess.as embedded embedded devices fashion inferences inputs language language understanding match microcontroller novel paper simple slu speech speech understanding spoken spoken language understanding temporal type understanding