Deletion Vectors in Delta Tables: Speeding Up Operations in Databricks

Traditionally, Delta Lake supports only the Copy-On-Write paradigm, in which underlying data files are changed anytime a file has been written. Example: When a single row in a file is deleted, the entire parquet file has to be rewritten. Considering your data is scattered among multiple files, and there are frequent updates to the data, this paradigm is not going to be efficient. For this kind of requirement, Databricks recently released a new feature named Deletion Vectors. In this article, …

azure databricks data engineering data warehouse