From Idea to Product: The Development of Apache Spark

Matei Zaharia, discusses the development of Spark from an idea to a full-fledged product. He began working on Spark as a PhD student, inspired by data center-scale computing happening at web companies like Google and Yahoo. Spark initially focused on addressing two use cases: machine learning with integrated algorithms and interactive ad hoc ways. Over time, Spark expanded to include other large-scale batch processing capabilities. He emphasizes the importance of finding people who actually want to do large-scale computing and …

algorithms apache apache spark cases center companies computing data data center development google interactive machine machine learning phd product scale spark use cases web yahoo