I tried scaling SageMaker Pipeline executions and this happened

Making your MLOps pipeline more resilient with retry policies

Yesterday I used SageMaker Pipelines to automate the workflow of a forecasting project. I launched 3 concurrent pipeline executions to train the model on different time horizons. Think predicting at 1 day, 1 week, 1 month.

After a while, 2 executions failed. It was a simple quota issue on training jobs.

Like everything in an MLOps project, your pipeline executions may fail due to a …

aws machine learning mlops pipeline sagemaker scaling