Parallelize your massive SHAP computations with MLlib and PySpark

A stepwise guide for efficiently explaining your models using SHAP.

Introduction to MLlib

Apache Spark’s Machine Learning Library (MLlib) is designed primarily for scalability and speed by leveraging the Spark runtime for common distributed use cases in supervised learning like classification and regression, unsupervised learning like clustering and collaborative filtering and in other cases like dimensionality reduction. In this article, I cover how we can use SHAP to explain a Gradient Boosted Trees (GBT) …

explainable ai machine learning massive mllib pyspark shap shapley-values spark-mllib