Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUs. (arXiv:2209.13443v1 [cs.LG])

With deep neural networks (DNNs) emerging as the backbone in a multitude of
computer vision tasks, their adoption in real-world consumer applications
broadens continuously. Given the abundance and omnipresence of smart devices,
"smart ecosystems" are being formed where sensing happens simultaneously rather
than standalone. This is shifting the on-device inference paradigm towards
deploying centralised neural processing units (NPUs) at the edge, where
multiple devices (e.g. in smart homes or autonomous vehicles) can stream their
data for processing with dynamic rates. …

arxiv batching edge exit networks neural networks