Machine Learning on a Large Scale

A demonstration using binomial and multinomial logistic regression in PySpark

With the release of Spark 3.2.1, that has been locally deployed for this article, PySpark offers a fluent API that resembles the expressivity of scikit-learn but additionally offers the benefits of distributed computing. This article demonstrates the use of the pyspark.ml module for constructing ML pipelines on top of Spark data frames (instead of RDDs with the older pyspark.mllib module). The functionality is exemplified …

learning logistic regression machine machine learning pyspark scale