Symphony: Optimized DNN Model Serving using Deferred Batch Scheduling

arXiv:2308.07470v2 Announce Type: replace-cross
Abstract: Having large batch sizes is one of the most critical aspects of increasing the accelerator efficiency and the performance of DNN model inference. However, existing model serving systems cannot achieve adequate batch sizes while meeting latency objectives as these systems eagerly dispatch requests to accelerators to minimize the accelerator idle time. We propose Symphony, a DNN serving system that explores deferred batch scheduling to optimize system efficiency and throughput. Further, unlike other prior systems, Symphony's …

abstract accelerator accelerators arxiv cs.dc cs.lg dnn efficiency inference latency performance scheduling systems type