How to develop a CD pipeline for Airflow

Leveraging GitHub actions and Astronomer to quickly push code updates to a production environment

Apache Airflow is a popular data orchestration tool used to manage workflows and tasks. However, one of the big questions I continue to come up against is how to deploy production-ready instances of Airflow. Options for hosting Airflow include self-management on a virtual machine, deploying to the cloud-based platform Astronomer, leveraging AWS MWAA, and more.

Of these options, I have …

airflow cd ci-cd-pipeline data engineering github pipeline python