AntBatchInfer: Elastic Batch Inference in the Kubernetes Cluster

arXiv:2404.09686v1 Announce Type: new
Abstract: Offline batch inference is a common task in the industry for deep learning applications, but it can be challenging to ensure stability and performance when dealing with large amounts of data and complicated inference pipelines. This paper demonstrated AntBatchInfer, an elastic batch inference framework, which is specially optimized for the non-dedicated cluster. AntBatchInfer addresses these challenges by providing multi-level fault-tolerant capabilities, enabling the stable execution of versatile and long-running inference tasks. It also improves inference …

abstract applications arxiv cluster cs.dc cs.lg data deep learning elastic framework industry inference kubernetes offline paper performance pipelines stability type