On Writing Clean Data Pipelines

Software Engineering for Data Science

Best practices for organizing your data analysis logic

When helping members of our community with architectural decisions regarding their data pipelines, a recurring question is how to group the analysis logic into tasks. This blog post summarizes the advice we’ve given to our community members for writing clean data pipelines. Our objective is to prevent an anti-pattern we’ve seen, where pipelines contain gigantic tasks:

These monolithic pipelines have many problems: …

data data engineering data pipelines data science jupyter-notebook machine learning open source pipelines writing