Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation. (arXiv:2108.12211v3 [cs.DC] UPDATED)

Distributed dataflow systems like Spark and Flink enable the use of clusters
for scalable data analytics. While runtime prediction models can be used to
initially select appropriate cluster resources given target runtimes, the
actual runtime performance of dataflow jobs depends on several factors and
varies over time. Yet, in many situations, dynamic scaling can be used to meet
formulated runtime targets despite significant performance variance.

This paper presents Enel, a novel dynamic scaling approach that uses message
propagation on an …

arxiv dataflow distributed graph jobs scaling