Shopify’s Practical Guidelines from Running Airflow for ML and Data Workflows at Scale

Shopify engineering shared its experience in the company's blog post on how to scale and optimize Apache Airflow for running ML and data workflows. They shared practical solutions for the challenges they faced like slow file access, insufficient control over DAG, irregular level of traffic, resource contention among workloads, and more.

ai airflow apache-airflow big data data data workflows devops guidelines machine learning ml ml & data engineering news scale shopify workflows