Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie

arXiv:2404.13682v1 Announce Type: cross
Abstract: As the Lakehouse architecture becomes more widespread, ensuring the reproducibility of data workloads over data lakes emerges as a crucial concern for data engineers. However, achieving reproducibility remains challenging. The size of data pipelines contributes to slow testing and iterations, while the intertwining of business logic and data management complicates debugging and increases error susceptibility. In this paper, we highlight recent advancements made at Bauplan in addressing this challenge. We introduce a system designed to …

abstract architecture arxiv cs.db cs.lg data data engineers data lakes data pipelines data science data workloads engineers however lakehouse lakehouse architecture pipelines reproducibility science testing type workloads