Idempotent writes to delta lake tables

Idempotent Writes to Delta Lake Tables

Walkthrough using open source delta lake

https://unsplash.com/photos/JI0KxozvOtQ

Introduction

According to Wikipedia:

Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.

Some of the non technical examples are elevator call buttons and crosswalk buttons. Having an idempotent software API is a critical required characteristic in many situations. One of such situations is Spark structured streaming. Structured streaming …

databricks data engineering data science delta delta-lake lake spark tables