Repetitive airflow pipeline problems

Hi r/datascience,

From my experience working with data orchestration tools (Airflow primarily), I tend to deal with a lot of repetitive fixes with flaky pipelines such as resource exhaustion issues, single malformed entries or other edge cases, figuring out why a task isn't running, and so on. I was wondering whether any of you had the same experience in your day-to-day work. How much of the job is actually just dealing with repetitive issues and maintenance of pipelines, and do …

airflow cases data datascience deal edge experience orchestration pipeline pipelines running tools