Data Transformation Using the Window Functions in PySpark

Demonstrated with a use case

Background

I work as an actuary in an insurance company. For various purposes we (securely) collect and store data for our policyholders in a data warehouse. One example is the claims payments data, for which large scale data transformations are required to obtain useful information for downstream actuarial analyses.

To demonstrate, one of the popular products we sell provides claims payment in the form of an income stream in …

data data science data transformation pyspark python transformation window-functions