MLOps for Batch Processing: Running Airflow on GPUs

A simple workaround for Apache Airflow limitations

There are a lot of MLOps platforms that natively deal with GPU access, but we enjoy the simplicity of Airflow for some of our tasks. This article looks at an approach that allows for both PyTorch and Tensorflow stacks to be used.

There are already a few articles about this type of implementation, but creating your own AWS AMI for Batch or having other approaches that …

airflow big data business strategy gpus hands-on-tutorials machine learning mlops processing