Performant IPv4 Range Spark Joins

A Practical guide to optimizing non-equi joins in Spark

Enriching network events with IP geolocation information is a crucial task, especially for organizations like the Canadian Centre for Cyber Security, the national CSIRT of Canada. In this article, we will demonstrate how to optimize Spark SQL joins, specifically focusing on scenarios involving non-equality conditions — a common challenge when working with IP geolocation data.

As cybersecurity practitioners, our reliance on enriching network events …

article canada centre cyber cyber security cybersecurity data engineering events geolocation guide hands-on-tutorials information ipv4 john joins lee network organizations practical security spark sql will