How to develop a CD pipeline for Airflow | allainews.com

Aug. 10, 2022, 1:34 p.m. | Chris Young

Towards Data Science - Medium towardsdatascience.com

Leveraging GitHub actions and Astronomer to quickly push code updates to a production environment

Photo by Mike Benna on Unsplash

Apache Airflow is a popular data orchestration tool used to manage workflows and tasks. However, one of the big questions I continue to come up against is how to deploy production-ready instances of Airflow. Options for hosting Airflow include self-management on a virtual machine, deploying to the cloud-based platform Astronomer, leveraging AWS MWAA, and more.

Of these options, I have …

airflow cd ci-cd-pipeline data engineering github pipeline python

More from towardsdatascience.com / Towards Data Science - Medium

Don’t Crash Your App: Load Records from the Database in Batches for Better Performance an hour ago | towardsdatascience.com

app coding data database +10

Create an AI-Driven Movie Quiz with Gemini LLM, Python, FastAPI, Pydantic, RAG and more an hour ago | towardsdatascience.com

artificial intelligence data engineering gemini hands-on-tutorials +1

Neural Speed: Fast Inference on CPU for 4-bit Large Language Models an hour ago | towardsdatascience.com

artificial intelligence cpp cpu data +14

Complex List Comprehensions Can Be Readable! 7 hours ago | towardsdatascience.com

data data science list list comprehension +6

Quantization, Linear Regression, and Hardware for AI: Our Best Recent Deep Dives 11 hours ago | towardsdatascience.com

data science deep-dives figure hardware +10

How do we know if AI is smoke and mirrors? 1 day, 1 hour ago | towardsdatascience.com

artificial intelligence deep-dives generative ai tools large language models +1

Feature Engineering with Microsoft Fabric and Dataflow Gen2 1 day, 2 hours ago | towardsdatascience.com

data engineering data science feature engineering microsoft fabric +1

How to Grow Your Career Without Feeling Stuck 1 day, 2 hours ago | towardsdatascience.com

boss career careers data +7

Callbacks and Pipeline structures in LangChain 1 day, 2 hours ago | towardsdatascience.com

basics callback cases create +12

Data Scientist (m/f/x/d)

@ Symanto Research GmbH & Co. KG | Spain, Germany

View on ai-jobs.net

Machine Learning Operations (MLOps) Engineer - Advisor

@ Peraton | Fort Lewis, WA, United States

View on ai-jobs.net

Mid +/Senior Data Engineer (AWS/GCP)

@ Capco | Poland

View on ai-jobs.net

Senior Software Engineer (ETL and Azure Databricks)|| RR/463/2024 || 4 - 7 Years

@ Emids | Bengaluru, India

View on ai-jobs.net

Senior Data Scientist (H/F)

@ Business & Decision | Toulouse, France

View on ai-jobs.net

Senior Analytics Engineer

@ Algolia | Paris, France

View on ai-jobs.net