BigDL 2.0: Seamless Scaling of AI Pipelines from Laptops to Distributed Cluster. (arXiv:2204.01715v1 [cs.LG])

Most AI projects start with a Python notebook running on a single laptop;
however, one usually needs to go through a mountain of pains to scale it to
handle larger dataset (for both experimentation and production deployment).
These usually entail many manual and error-prone steps for the data scientists
to fully take advantage of the available hardware resources (e.g., SIMD
instructions, multi-processing, quantization, memory allocation optimization,
data partitioning, distributed computing, etc.). To address this challenge, we
have open sourced BigDL …

ai arxiv cluster distributed laptops pipelines scaling